hadoop的HA配置（高可用配置）_网页overview 'master:8020' (standby)

一、修改配置文件（注意，一定设置免密登陆）

直接上配置文件吧。在etc下，修改hdfs-site.xml，
hosts文件
192.168.3.201 service1
192.168.3.202 service2
192.168.3.203 service3
192.168.3.204 service4
192.168.3.205 service5
192.168.3.206 service6
192.168.3.207 service7
192.168.3.208 service8
192.168.3.209 service9
192.168.3.210 service10

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
       Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>service6:50090</value>
    </property>
    <property>
        <name>dfs.nameservices</name>
        <value>mycluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.mycluster</name>
        <value>nn1,nn2</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn1</name>
        <value>service1:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn2</name>
        <value>service6:8020</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn1</name>
        <value>service1:50070</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn2</name>
        <value>service6:50070</value>
    </property>
    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://service2:8485;service3:8485;service4:8485/mycluster</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.mycluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>
                sshfence
                shell(/bin/true)
        </value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_rsa</value>
    </property>
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/zdq/hadoop-2.7.4/journal</value>
    </property>
</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

修改：core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/zdq/hadoop-2.7.4/tmp</value>
        </property>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://mycluster/</value>
        </property>
</configuration>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

二、发送配置文件到各个主机。我这里用了

在service2、service3、service4启动journalnode

$>hadoop-daemon.sh start journalnode
1

确定三台都启动了JournalNode进程。使用jps命令即可看见

三、拷贝数据到新的NN节点。

$>scp -r /zdq/hadoop-2.7.4/tmp/dfs root@service6:/zdq/hadoop-2.7.4/tmp/
1

如果不执行以上操作，可能出现两个集群版本不一致，导致最后加入到一个集群失败

四、启动NN节点。

$>hadoop-daemon.sh start namenode
1
2

五、在新的节点上实行状态引导(我这里新节点是service6)

$>hdfs namenode -bootstrapStandby
1

提示是否格式化，选择N，不要格式化。

六、在一个NN上执行，完成edit日志传输到JN

$>hdfs namenode -initializeSharedEdits
1

七、启动DN和新的NN启动

//启动namenode
$>hadoop-daemon.sh start namenode

//启动datanode
$>hadoop-daemons.sh start datanode
1
2
3
4
5

这时候查看50070端口应该是能查看到了。
不过在网页上都是看到
Overview ‘service1:8020’ (standby)
Overview ‘service6:8020’ (standby)
standby：待命状态
Active：激活状态
所以我要激活其中一个节点

$>hdfs haadmin -transitionToActive nn1
1

然后看到的效果应该是：
Overview ‘service1:8020’ (active)
Overview ‘service6:8020’ (standby)

八、配置完成，附加说明

//转换成激活态，nn1提供服务
$>hdfs haadmin -transitionToActive nn1
//转换层待命态
$>hdfs haadmin -transitionToStandby nn1
//容灾nn1到nn2，也就是nn2提供服务
$>hdfs haadmin -failover nn1 nn2
1
2
3
4
5
6