大数据学习总结（2021版）---Zookeeper_大数据zookeep的实训心得

第1章 Zookeeper入门

---

1.1 特点：

• 1）Zookeeper：一个领导者（leader），多个跟随者（follower）组成的集群。
• 2）Leader负责进行投票的发起和决议，更新系统状态
• 3）Follower用于接收客户请求并向客户端返回结果，在选举Leader过程中参与投票
• 4）集群中只要有半数以上节点存活，Zookeeper集群就能正常服务
• 5）全局数据一致：每个server保存一份相同的数据副本，client无论连接到哪个server，数据都是一致的。
• 6）更新请求顺序进行，来自同一个client的更新请求按其发送顺序依次执行。
• 7）数据更新原子性，一次数据更新要么成功，要么失败。
• 8）实时性，在一定时间范围内，client能读到最新数据。

---

1.2 应用场景

• 统一命名服务

• 统一配管理

将配置信息写入ZooKeeper上的Znode,各个客户端服务器监听这个Znode，一旦Znode中数据被修改，ZooKeeper将通知各个客户端服务器

• 统一集群管理

实时监控节点状态变化

• 服务器节点动态上下线

客户端实时洞察到服务器上下线变化

• 软负载均衡

Zookeeper 中记录每台服务器的访问数，让访问数最少的服务器去处理最新客户端请求

第3章 Zookeeper内部原理

3.1 选举机制（面试重点）

• （1）半数机制：集群中半数以上机器存活，集群可用。所以Zookeeper适合安装奇数台服务器。
• （2）Zookeeper工作时，是有一个节点为Leader，其他则为Follower，Leader是通过内部的选举机制临时产生的。
• （3）选举流程：理解！！！！

3.2 节点类型（持久、非持久、有无顺序编号）

• 持久：（客户端与Zookeeper断开后，节点依旧存在）
• 短暂：（客户端与Zookeeper断开后，节点被删除）

节点的创建（持久、非持久、有无顺序）方式，下面会讲到

3.3 监听器原理（面试重点）

• main（）线程：connect、listener
• connect将监听事件发送给Zookeeper
• Zookeeper将监听事件添加到列表
• Zookeeper监听到数据变化或路径变化，将消息发送给listener
• listener线程内部调用process方法

3.4 写数据流程

第4章 Zookeeper实战（开发重点）

4.1 分布式安装部署

---

1．集群规划
在hadoop102、hadoop103和hadoop104三个节点上部署Zookeeper

---

2．解压安装
（1）解压Zookeeper安装包到/opt/module/目录下
（2）同步/opt/module/zookeeper-3.4.10目录内容到hadoop103、hadoop104

---

3．配置zoo.cfg文件

• （1）将/opt/module/zookeeper-3.4.10/conf这个路径下的zoo_sample.cfg修改为zoo.cfg；
• （2）打开zoo.cfg文件，修改dataDir路径：

修改成：dataDir=/opt/module/zookeeper-3.4.10/zkData

• （3）打开zoo.cfg文件，增加如下配置
  

#######################cluster##########################
server.2=hadoop102:2888:3888
server.3=hadoop103:2888:3888
server.4=hadoop104:2888:3888
1
2
3
4

• （4）同步zoo.cfg配置文件
• （5）配置参数解读

---

4．配置服务器编号
（1）在/opt/module/zookeeper-3.4.10/这个目录下创建zkData

[atguigu@hadoop102 zookeeper-3.4.10]$ mkdir -p zkData
1

（2）在/opt/module/zookeeper-3.4.10/zkData目录下创建一个myid的文件

（3）编辑myid文件
（4）拷贝配置好的zookeeper到其他机器上（拷贝好后，需要去103,104x修改myid为3,4）

---

5．集群操作
（1）分别启动Zookeeper

[atguigu@hadoop102 zookeeper-3.4.10]$ bin/zkServer.sh start
[atguigu@hadoop103 zookeeper-3.4.10]$ bin/zkServer.sh start
[atguigu@hadoop104 zookeeper-3.4.10]$ bin/zkServer.sh start
1
2
3

（2）查看状态

[atguigu@hadoop102 zookeeper-3.4.10]# bin/zkServer.sh status
1

4.2 客户端命令行操作

1．启动客户端

[atguigu@hadoop103 zookeeper-3.4.10]$ bin/zkCli.sh
1

---

2．显示所有操作命令

[zk: localhost:2181(CONNECTED) 1] help
1

---

3．查看当前znode中所包含的内容

[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
1
2

---

4．查看当前节点详细数据

[zk: localhost:2181(CONNECTED) 1] ls2 /
1

---

5．分别创建2个普通节点：create

---

6．获得节点的值：get

[zk: localhost:2181(CONNECTED) 5] get /sanguo
1

---

7．创建短暂节点：-e

[zk: localhost:2181(CONNECTED) 7] create -e /sanguo/wuguo "zhouyu"
1

---

8．创建带序号的节点：-s

---

9．修改节点数据值：set

[zk: localhost:2181(CONNECTED) 6] set /sanguo/weiguo "simayi"
1

---

10．节点的值变化监听：watch

---

11．删除节点：delete

[zk: localhost:2181(CONNECTED) 4] delete /sanguo/jin
1

---

12．递归删除节点：rmr

[zk: localhost:2181(CONNECTED) 15] rmr /sanguo/shuguo
1

---

13．查看节点状态：stat

[zk: localhost:2181(CONNECTED) 17] stat /sanguo
1

4.3 API应用

4.3.1 Eclipse环境搭建

1．创建一个Maven工程
2．添加pom文件

<dependencies>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>RELEASE</version>
		</dependency>
		<dependency>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-core</artifactId>
			<version>2.8.2</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper -->
		<dependency>
			<groupId>org.apache.zookeeper</groupId>
			<artifactId>zookeeper</artifactId>
			<version>3.4.10</version>
		</dependency>
</dependencies>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

3．拷贝log4j.properties文件到项目根目录
需要在项目的src/main/resources目录下，新建一个文件，命名为“log4j.properties”，在文件中填入。

log4j.rootLogger=INFO, stdout  
log4j.appender.stdout=org.apache.log4j.ConsoleAppender  
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout  
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n  
log4j.appender.logfile=org.apache.log4j.FileAppender  
log4j.appender.logfile.File=target/spring.log  
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout  
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n  
1
2
3
4
5
6
7
8

4.3.2 创建ZooKeeper客户端

package com.atguigu.com;

import java.io.IOException;
import java.util.List;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.junit.Before;
import org.junit.Test;

public class TestZookeeper {

    private String connectString="hadoop102:2181,hadoop103:2181,hadoop104:2181";
    private int sessionTimeout = 2000;
    private ZooKeeper zkClient;

    @Before
    public void init() throws IOException{

        zkClient = new ZooKeeper(connectString, sessionTimeout , new Watcher() {

            @Override
            public void process(WatchedEvent event) {

				System.out.println("---------start----------");
				List<String> children;
				try {
					children = zkClient.getChildren("/", true);

					for (String child : children) {
						System.out.println(child);
					}
					System.out.println("---------end----------");
				} catch (KeeperException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				} catch (InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
            }
        });
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

4.3.3 创建子节点

package com.atguigu.com;

import java.io.IOException;
import java.util.List;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.junit.Before;
import org.junit.Test;

public class TestZookeeper {

    private String connectString="hadoop102:2181,hadoop103:2181,hadoop104:2181";
    private int sessionTimeout = 2000;
    private ZooKeeper zkClient;

    @Before
    public void init() throws IOException{

        zkClient = new ZooKeeper(connectString, sessionTimeout , new Watcher() {

            @Override
            public void process(WatchedEvent event) {

				System.out.println("---------start----------");
				List<String> children;
				try {
					children = zkClient.getChildren("/", true);

					for (String child : children) {
						System.out.println(child);
					}
					System.out.println("---------end----------");
				} catch (KeeperException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				} catch (InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
            }
        });
    }

    // 1 创建节点
    @Test
    public void createNode() throws KeeperException, InterruptedException{

        String path = zkClient.create("/sunyl", "dahaigezuishuai".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

        System.out.println(path);

    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

4.3.4 获取子节点并监听节点变化

package com.atguigu.com;

import java.io.IOException;
import java.util.List;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.junit.Before;
import org.junit.Test;

public class TestZookeeper {

    private String connectString="hadoop102:2181,hadoop103:2181,hadoop104:2181";
    private int sessionTimeout = 2000;
    private ZooKeeper zkClient;

    @Before
    public void init() throws IOException{

        zkClient = new ZooKeeper(connectString, sessionTimeout , new Watcher() {

            @Override
            public void process(WatchedEvent event) {

				System.out.println("---------start----------");
				List<String> children;
				try {
					children = zkClient.getChildren("/", true);

					for (String child : children) {
						System.out.println(child);
					}
					System.out.println("---------end----------");
				} catch (KeeperException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				} catch (InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
            }
        });
    }

    // 2 获取子节点 并监控节点的变化
    @Test
    public void getDataAndWatch() throws KeeperException, InterruptedException{

        List<String> children = zkClient.getChildren("/", true);

        for (String child : children) {
            System.out.println(child);
        }

        Thread.sleep(Long.MAX_VALUE);
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

4.3.5 判断Znode是否存在

package com.atguigu.com;

import java.io.IOException;
import java.util.List;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.junit.Before;
import org.junit.Test;

public class TestZookeeper {

    private String connectString="hadoop102:2181,hadoop103:2181,hadoop104:2181";
    private int sessionTimeout = 2000;
    private ZooKeeper zkClient;

    @Before
    public void init() throws IOException{

        zkClient = new ZooKeeper(connectString, sessionTimeout , new Watcher() {

            @Override
            public void process(WatchedEvent event) {

				System.out.println("---------start----------");
				List<String> children;
				try {
					children = zkClient.getChildren("/", true);

					for (String child : children) {
						System.out.println(child);
					}
					System.out.println("---------end----------");
				} catch (KeeperException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				} catch (InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
            }
        });
    }


    // 3 判断节点是否存在
    @Test
    public void exist() throws KeeperException, InterruptedException{

        Stat stat = zkClient.exists("/sunyl", false);

        System.out.println(stat==null? "not exist":"exist");
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

4.4 监听服务器节点动态上下线案例

1．需求
某分布式系统中，主节点可以有多台，可以动态上下线，任意一台客户端都能实时感知到主节点服务器的上下线。

---

2．具体实现

• （0）先在集群上创建/servers节点

[zk: localhost:2181(CONNECTED) 1] create /servers "servers"
Created /servers
1
2

• （1）服务器端向Zookeeper注册代码

package com.atguigu;

import java.io.IOException;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.ZooDefs.Ids;

public class DistributeServer {

	public static void main(String[] args) throws Exception {
		
		DistributeServer server = new DistributeServer();
		
		// 1 连接zookeeper集群
		server.getConnect();
		
		// 2 注册节点
		server.regist(args[0]);
		
		// 3 业务逻辑处理
		server.business();
	}

	private void business() throws InterruptedException {
	
		Thread.sleep(Long.MAX_VALUE);
	}

	private void regist(String hostname) throws KeeperException, InterruptedException {
		
		String path = zkClient.create("/servers/server", hostname.getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
		
		System.out.println(hostname +" is online ");
		
	}

	private String connectString = "hadoop102:2181,hadoop103:2181,hadoop104:2181";
	private int sessionTimeout = 2000;
	private ZooKeeper zkClient;

	private void getConnect() throws IOException {
		
		zkClient = new ZooKeeper(connectString , sessionTimeout , new Watcher() {
			
			@Override
			public void process(WatchedEvent event) {
				// TODO Auto-generated method stub
				
			}
		});
	}
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

测试：注册hadoop104节点


此时客户端也会实时监控到

• （2）客户端代码

package com.atguigu;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;

public class DistributeClient {

	public static void main(String[] args) throws IOException, KeeperException, InterruptedException {
		
		DistributeClient client = new DistributeClient();
		
		// 1 获取zookeeper集群连接
		client.getConnect();
		
		// 2 注册监听
		client.getChlidren();
		
		// 3 业务逻辑处理
		client.business();
		
	}

	private void business() throws InterruptedException {
		Thread.sleep(Long.MAX_VALUE);
	}

	private void getChlidren() throws KeeperException, InterruptedException {
		
		List<String> children = zkClient.getChildren("/servers", true);
		
		// 存储服务器节点主机名称集合
		ArrayList<String> hosts = new ArrayList<>();
		
		for (String child : children) {
			
			byte[] data = zkClient.getData("/servers/"+child, false, null);
			
			hosts.add(new String(data));
		}
		
		// 将所有在线主机名称打印到控制台
		System.out.println(hosts);
		
	}

	private String connectString = "hadoop102:2181,hadoop103:2181,hadoop104:2181";
	private int sessionTimeout = 2000;
	private ZooKeeper zkClient;

	private void getConnect() throws IOException {
	
		zkClient = new ZooKeeper(connectString , sessionTimeout , new Watcher() {
			
			@Override
			public void process(WatchedEvent event) {
				
				try {
					getChlidren();
				} catch (KeeperException e) {
					e.printStackTrace();
				} catch (InterruptedException e) {
					e.printStackTrace();
				}
			}
		});
		
	}
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

第5章 企业面试真题

5.1 请简述ZooKeeper的选举机制

详见3.1。

5.2 ZooKeeper的监听原理是什么？

详见3.4。

5.3 ZooKeeper的部署方式有哪几种？集群中的角色有哪些？集群最少需要几台机器？

（1）部署方式单机模式、集群模式
（2）角色：Leader和Follower
（3）集群最少需要机器数：3

5.4 ZooKeeper的常用命令

ls create get delete set…