部署hdfs_hdfs部署在哪里

hadoop三大组件：

1、hdfs
2、mapreduce
3、yarn
我们要求：hdfs三个进程要以hadoop001启动

hadoop部署文件目录：/home/ali/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

1、有些版本的是在conf目录中
2、在这个目录下有*.cmd文件，这些文件是windows部署文件，在linux的机器上可以删掉啦

主要文件：

core-site.xml：把hadoop三组件的核心配置单独放一个配置文件
hadoop-env.sh：配置了JDK目录、hadoop家目录
hdfs-site.xml：hdfs配置文件
mapred-site.xml.template：mapreduce配置文件
yarn-site.xml：yarn配置文件
slaves：下文用来配置了datanode的机器
log4j.properties

修改机器名：

1、注意不要用ip部署，统一机器名称hostname部署（否则网段变化就要改变部署）

永久修改机器名，需要重启机器才能生效

[root@localhost /]# vi /etc/sysconfig/network

# Created by anaconda
NETWORKING=yes
HOSTNAME=hadoop001
1
2
3
4
5

这有点麻烦，所以我们在临时修改一下机器名，立即生效

临时修改机器名

[root@localhost /]# hostname hadoop001

#查询机器名
[root@localhost /]# uname -n
hadoop001
1
2
3
4
5

2、当网段发生改变只需要/etc/hosts 修改即可，做好机器名和ip的映射

[root@localhost /]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.248.223 hadoop001
1
2
3
4

注意：第一行和第二行不要删除，文件本身的内容不要删除

hdfs配置

1、修改namenode进程配置:

[hadoop@hadoop001 hadoop]$ vi core-site.xml 

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop001:9000</value>
    </property>
</configuration>
1
2
3
4
5
6
7
8

2、修改datanode进程配置:
可是配置多个datanode，以逗号分隔

[hadoop@hadoop001 hadoop]$ vi slaves 
hadoop001,hadoop002,hadoop003
1
2

3、修改secondarynamenode进程配置:

[hadoop@hadoop001 hadoop]$ vi hdfs-site.xml
<property>
	<name>dfs.namenode.secondary.http-address</name>
	<value>hadoop001:50090</value>
</property>
<property>
	<name>dfs.namenode.secondary.https-address</name>
	<value>hadoop001:50091</value>
</property>
1
2
3
4
5
6
7
8
9

4、认证机器名hadoop001，否则会被要求输入密码

[ali@hadoop001 ~]$ rm -rf .ssh
[ali@hadoop001 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ali/.ssh/id_rsa):
Created directory '/home/ali/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ali/.ssh/id_rsa.
Your public key has been saved in /home/ali/.ssh/id_rsa.pub.
The key fingerprint is:
6b:06:a8:62:c0:aa:67:6b:3d:f0:f2:fe:78:15:2a:f1 ali@hadoop001
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|.   ..  .        |
|..  .o..S.       |
|o ... E...       |
|o..+ . .+        |
|o.= +..o         |
|.+.=++.          |
+-----------------+
[ali@hadoop001 ~]$ cd .ssh
[ali@hadoop001 .ssh]$ ll
总用量 8
-rw-------. 1 ali ali 1675 4月  30 03:36 id_rsa
-rw-r--r--. 1 ali ali  395 4月  30 03:36 id_rsa.pub
[ali@hadoop001 .ssh]$ cat id_rsa.pub >> authorized_keys
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

还是要输入密码？？authorized_keys权限问题

chmod 600 authorized_keys
修改之后就不需要再输入密码了！！
修改步骤如下：

[ali@hadoop001 .ssh]$ chmod 600 authorized_keys
1

启动hdfs，发现三个进程都以hadoop001启动，完美

[ali@hadoop001 sbin]$ ./start-dfs.sh
19/04/30 03:44:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop001]
hadoop001: starting namenode, logging to /home/ali/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-ali-namenode-hadoop001.out
hadoop001: starting datanode, logging to /home/ali/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-ali-datanode-hadoop001.out
Starting secondary namenodes [hadoop001]
hadoop001: starting secondarynamenode, logging to /home/ali/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-ali-secondarynamenode-hadoop001.out
19/04/30 03:45:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1
2
3
4
5
6
7
8

好的，hdfs就配置完了。

localhost:50070可以访问hdfs的web界面。