在Centos7中安装Hadoop详细教程_centos7安装hadoop

系统要求

系统：Centos7
Hadoop版本：2.8.0

安装JAVA环境

Hadoop许多组件都是JAVA语言编写的，安装JAVA环境才能正常运行，所以JAVA环境必不可少。
JAVA版本：1.8.0_65
百度网盘下载地址：
链接：https://pan.baidu.com/s/1gtwyeF9UVnMfnb5x3bQNKw
提取码：7mk5

可使用WINSCP 文件上传至Centos 7Linux

使用命令：
tar -zxvf jdk-8u65-linux-x64.t.gz

[root@zhang jvm]# tar -zxvf jdk-8u65-linux-x64.t.gz
jdk1.8.0_65
1
2

进入刚上传的路径，输入./java -version查询版本，显示如下，表示安装成功

cd /usr/lib/jvm/jdk1.8.0_65/bin
[root@zhang bin]# ./java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
1
2
3
4
5

配置环境变量PATH（重要）
使用VIM打开，编辑profile

vi /etc/profile
1

在末端添加以下代码，如果跟我路径一致，可直接复制，不一样的话，要注意自己安装路径，即上面上传解压的路径。就是这一句代码修改为自己的路径：export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_65，后面2个直接复制即可。

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_65
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
1
2
3

使用wq!保存退出
修改完成要使环境变量生效，要source下

[root@zhang bin]# source /etc/profile
[root@zhang bin]#
1
2

测试下环境变量是否起作用，能显示版本号就成功

[root@zhang bin]# java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
[root@zhang bin]# 
1
2
3
4
5

安装Hadoop的HDFS和Yarn

Hadoop版本号：hadoop-2.8.0
网盘下载地址：
链接：https://pan.baidu.com/s/1QJ8x8NJGkV4j6WD-JiNJ6Q
提取码：xopm
使用WINSCP上传至Centos 7 Linux系统

选择上传目标路径，我这里选择:/opt/tools

[root@zhang bin]# cd /opt/tools/
[root@zhang tools]# ls
apache-flume-1.9.0-bin.tar.gz  hadoop-2.8.0.tar.gz

1
2
3
4

解压

tar -zxvf hadoop-2.8.0.tar.gz
1

使用CP 命令复制解压后的文件到 /opt/server/,如下复制后并更名为hadoop-2.8.0

[root@zhang server]# ls
flume  hadoop-2.8.0
1
2

Hadoop安装路径如下：后面设置环境变量需要，要记住这个地址

[root@zhang server]# cd hadoop-2.8.0/
[root@zhang hadoop-2.8.0]# ls
bin   etc      lib      LICENSE.txt  NOTICE.txt  sbin
data  include  libexec  logs         README.txt  share
[root@zhang hadoop-2.8.0]# pwd
/opt/server/hadoop-2.8.0
1
2
3
4
5
6

配置hadoop-env.sh，末尾增添JAVA_HOME 的路径

root@zhang hadoop]# vi /opt/server/hadoop-2.8.0/etc/hadoop/hadoop-env.sh 
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_65
1
2

配置hadoop-env.sh，末尾增添JAVA_HOME 的路径

配置 core-site.xml
在配置文件夹/opt/server/hadoop-2.8.0/etc/hadoop/
使用VIM编辑

vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://zhang:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/server/hadoop-2.8.0/data/</value>
</property>
</configuration>
1
2
3
4
5
6
7
8
9
10
11

注意：
第一个property中的Value:hdfs://zhang:9000/，修改主机名，比如我的主机名为zhang,端口号9000是固定的，特别注意下 主机名，可以使用
hostname 查询下，错了容易出现问题

第二个property中的Value:hadoop的安装路径，opt/server/hadoop-2.8.0/data/，数据存储在data文件夹里

配置 hdfs-site.xml

<configuration>
<property>
　　　　　　　　<name>dfs.replication</name>
　　　　　　　　<value>1</value>
　　　　　　</property>
<property>
</configuration>
1
2
3
4
5
6
7

配置 mapred-site.xml
打开mapred-site.xml,编辑以下内容，直接复制

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
1
2
3
4
5
6

配置 yarn-site.xml
打开yarn-site.xml,编辑以下内容，直接复制

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>zhang</value>
</property>

<property>
<name>yarn.nodemanager.aux-servies</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

1
2
3
4
5
6
7
8
9
10
11
12

第一个property的value为主机名，要更改为自己的主机名，另一个直接复制，无需更改

配置Hadoop环境变量，跟JAVA环境变量一致，只需在profile文件末尾添加即可

vi /etc/profile
export HADOOP_HOME=/opt/server/hadoop-2.8.0
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
1
2
3
4
5
6
7
8

用户名可以按实际写，Hadoop_Home 为安装路径，家目录，根据实际路径填写即可。

使用SSH免密登录

设置SSH，可以免密操作，无需频繁输入密码，尤其是多个节点操作时需要免密登录

ssh-keygen -t rsa
1

一路ENTER确认即可

[root@zhang hadoop]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Hf31D+MXPXuDrXUG8AKWri+FxKVWTozOPchnmn/s6NA root@zhang
The key's randomart image is:
+---[RSA 2048]----+
|         o       |
|        . =o     |
|       = O= o   .|
|        X+*o + .o|
|       oS*o.. *oo|
|        +o.  oo+=|
|        ooE. ..=*|
|        .o..o oo+|
|         o+o..   |
+----[SHA256]-----+
[root@zhang hadoop]# 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

将公钥追加到authorized_keys文件中去

[root@zhang hadoop]# cd /root/.ssh/
[root@zhang .ssh]# ls
authorized_keys  id_rsa  id_rsa.pub  known_hosts
[root@zhang .ssh]# 
[root@zhang .ssh]# cat id_rsa.pub >> authorized_keys
[root@zhang .ssh]# cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAgPnSvMuak7tITg4osi1paBHZracOfD5OVvgWnIocmv2p+ShYv4GCjGXEEJRJPRzAf+6J25lTEaQQbZf6pFiS72lYbZltPQ8E2Mb00wFE/cuyJLoI49dgAISTF+1PYZqcuFIbPPM+voZYPp+8HxJqXkBZj7XHvxPDNrU2k13lNrRsgvLEXRHVgRIjJaZIar9yx6Tm21XyY88Eqnw0WhoIFw/TqTMEu7dYcBKNfgoDTRcg/xTyDqAGpZOxiYHtGNqp7F3R2ngw5QbzBo0O3MrsifX9jmSeueDMWk92xIDR1cXPYwrArXASwFjQBszl8aMx9XZ8a+Rs1Xjt2DQ0VRzb root@localhost.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAisWpCpt9fK6NcMKEKz2eD1yP6BEOOrDjWcQhWXjmuHjhrbVsjZEUf6xr/ub3k/21Q0QeoFfK64fg7nF1aukS9jxNB1L8uBt5f95YnUPiocz5cLp7XzkMf4qInP5aXCzLkHFvvi7NThYsrfp3yUGMKt/8VgJSXVJTCPssG2+sMBkevZSDIFm5o1BKUmMuQWu82T7g4GXOw3F0y+ZrHM+oniF1fliaSNj7r1wUM8YgKxeHYrLn9JdkH/eil7Ay777MZGfO1zLDwsJjQVhiEY6cnsFKDy4nAutI/ep6nr+o0dCPLrUkSAfSPa2JFFb3F9YJN3GJ0jNZjgfqQRinNRGX root@localhost.localdomain
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN88B3ux29tH0qRQbaXM5bpEEmqRusQ2gbNEaSxLGeONJeG78cUnkmFeLdgKznZI99sDwhlhwL4jnrL8jWvOou+II6KcRJ6k45myDpg9GNqQvc836r7v4hMetIAeHjtRAOZjIT7CuT+o+9BviF7xrUerC9vyXn+8lUfNEZW+TnmTaV26bVxeET/RX13z8em5wJzWRYRtSXCVBB3cthpwBm7oxzoXdOj/gtGWpq5bg3iDgeEPWSwor1//Jh+hlYTCvvfTpCpX9QLuB+WoWTX5nsUgqd5FDME2tsPFQCoAstr17iFgUyr4fhuVU7/emC+cbYzqK15Iey4o1UK0ghNWAH root@zhang
[root@zhang .ssh]# 
1
2
3
4
5
6
7
8
9
10

使用SSH 主机名 登录，无需密码

[root@zhang .ssh]# ssh zhang
Last login: Tue Jul 16 12:40:56 2024
[root@zhang ~]# 
1
2
3

启动HADOOP

初始化
这一步很重要
如果出现 INFO common.Storage: Storage directory /opt/server/hadoop-2.8.0/data/dfs/name has been successfully formatted. 为成功格式化

[root@zhang ~]# hadoop namenode -format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

24/07/16 13:00:59 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   user = root
STARTUP_MSG:   host = zhang/192.168.66.128
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.8.0
STARTUP_MSG:   classpath = /opt/server/hadoop-2.8.0/etc/hadoop:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/activation-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/avro-1.7.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/commons-net-3.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/gson-2.2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/guava-11.0.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/hadoop-annotations-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/hadoop-auth-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jcip-annotations-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jettison-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jsch-0.1.51.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/json-smart-1.1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/junit-4.11.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/paranamer-2.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/xz-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/hadoop-common-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/hadoop-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/common/hadoop-nfs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/okio-1.4.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-client-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/activation-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/commons-math-2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/fst-2.24.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/guice-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/objenesis-2.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/xz-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-api-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-client-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-registry-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.0-tests.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.0.jar:/opt/server/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar:/opt/server/hadoop-2.8.0/contrib/capacity-scheduler/*.jar:/opt/server/hadoop-2.8.0/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 91f2b7a13d1e97be65db92ddabc627cc29ac0009; compiled by 'jdu' on 2017-03-17T04:12Z
STARTUP_MSG:   java = 1.8.0_65
************************************************************/
24/07/16 13:00:59 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
24/07/16 13:00:59 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-8b3bc4bb-baa8-4fb4-b71a-9a26a6626ef5
24/07/16 13:01:01 INFO namenode.FSEditLog: Edit logging is async:false
24/07/16 13:01:01 INFO namenode.FSNamesystem: KeyProvider: null
24/07/16 13:01:01 INFO namenode.FSNamesystem: fsLock is fair: true
24/07/16 13:01:01 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
24/07/16 13:01:02 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
24/07/16 13:01:02 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
24/07/16 13:01:02 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
24/07/16 13:01:02 INFO blockmanagement.BlockManager: The block deletion will start around 2024 Jul 16 13:01:02
24/07/16 13:01:02 INFO util.GSet: Computing capacity for map BlocksMap
24/07/16 13:01:02 INFO util.GSet: VM type       = 64-bit
24/07/16 13:01:02 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
24/07/16 13:01:02 INFO util.GSet: capacity      = 2^21 = 2097152 entries
24/07/16 13:01:02 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
24/07/16 13:01:02 INFO blockmanagement.BlockManager: defaultReplication         = 1
24/07/16 13:01:02 INFO blockmanagement.BlockManager: maxReplication             = 512
24/07/16 13:01:02 INFO blockmanagement.BlockManager: minReplication             = 1
24/07/16 13:01:02 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
24/07/16 13:01:02 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
24/07/16 13:01:02 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
24/07/16 13:01:02 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
24/07/16 13:01:02 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
24/07/16 13:01:02 INFO namenode.FSNamesystem: supergroup          = supergroup
24/07/16 13:01:02 INFO namenode.FSNamesystem: isPermissionEnabled = true
24/07/16 13:01:02 INFO namenode.FSNamesystem: HA Enabled: false
24/07/16 13:01:02 INFO namenode.FSNamesystem: Append Enabled: true
24/07/16 13:01:02 INFO util.GSet: Computing capacity for map INodeMap
24/07/16 13:01:02 INFO util.GSet: VM type       = 64-bit
24/07/16 13:01:02 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
24/07/16 13:01:02 INFO util.GSet: capacity      = 2^20 = 1048576 entries
24/07/16 13:01:02 INFO namenode.FSDirectory: ACLs enabled? false
24/07/16 13:01:02 INFO namenode.FSDirectory: XAttrs enabled? true
24/07/16 13:01:02 INFO namenode.NameNode: Caching file names occurring more than 10 times
24/07/16 13:01:02 INFO util.GSet: Computing capacity for map cachedBlocks
24/07/16 13:01:02 INFO util.GSet: VM type       = 64-bit
24/07/16 13:01:02 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
24/07/16 13:01:02 INFO util.GSet: capacity      = 2^18 = 262144 entries
24/07/16 13:01:02 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
24/07/16 13:01:02 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
24/07/16 13:01:02 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
24/07/16 13:01:02 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
24/07/16 13:01:02 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
24/07/16 13:01:02 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
24/07/16 13:01:03 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
24/07/16 13:01:03 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
24/07/16 13:01:03 INFO util.GSet: Computing capacity for map NameNodeRetryCache
24/07/16 13:01:03 INFO util.GSet: VM type       = 64-bit
24/07/16 13:01:03 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
24/07/16 13:01:03 INFO util.GSet: capacity      = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /opt/server/hadoop-2.8.0/data/dfs/name ? (Y or N) y
24/07/16 13:01:06 INFO namenode.FSImage: Allocated new BlockPoolId: BP-108476542-192.168.66.128-1721106066849
24/07/16 13:01:06 INFO common.Storage: Storage directory /opt/server/hadoop-2.8.0/data/dfs/name has been successfully formatted.
24/07/16 13:01:06 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/server/hadoop-2.8.0/data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
24/07/16 13:01:07 INFO namenode.FSImageFormatProtobuf: Image file /opt/server/hadoop-2.8.0/data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.
24/07/16 13:01:07 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
24/07/16 13:01:07 INFO util.ExitUtil: Exiting with status 0
24/07/16 13:01:07 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at zhang/192.168.66.128
************************************************************/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

启动进程
start-dfs.sh
start-yarn.sh

[root@zhang hadoop]# start-dfs.sh
Starting namenodes on [zhang]
zhang: starting namenode, logging to /opt/server/hadoop-2.8.0/logs/hadoop-root-namenode-zhang.out
localhost: starting datanode, logging to /opt/server/hadoop-2.8.0/logs/hadoop-root-datanode-zhang.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/server/hadoop-2.8.0/logs/hadoop-root-secondarynamenode-zhang.out
[root@zhang hadoop]# 
1
2
3
4
5
6
7

[root@zhang hadoop]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/server/hadoop-2.8.0/logs/yarn-root-resourcemanager-zhang.out
localhost: starting nodemanager, logging to /opt/server/hadoop-2.8.0/logs/yarn-root-nodemanager-zhang.out
[root@zhang hadoop]# 
1
2
3
4
5

关闭Hadoop

stop-all.sh
1

也可以

stop-yarn.sh
start-dfs.sh
1
2

如果出现namenode 或datanode无法启动，删除/opt/server/hadoop-2.8.0/l所有version文件

验证安装结果

使用jps命令，显示如下6个进程即安装成功

[root@zhang hadoop]# jps
60034 NodeManager
59907 ResourceManager
59654 SecondaryNameNode
59466 DataNode
60618 Jps
59307 NameNode
[root@zhang hadoop]# 
1
2
3
4
5
6
7
8

最后可以应用flume采集日志数据，实时地传输到这个Hadoop文件系统，再次验证这个系统的安装是否准确无误，可以参考我其他文章，如有问题，欢迎一起学习讨论！
Flume+Hadoop大数据采集部署

参考文章
在Linux虚拟机下安装Hadoop的保姆级教程