chaosblade混沌测试_chaosblade box部署

ChaosBlade 是阿里巴巴 2019 年开源的混沌工程项目，包含混沌工程实验工具 chaosblade 和混沌工程平台 chaosblade-box，旨在通过混沌工程帮助企业解决云原生过程中高可用问题。实验工具 chaosblade 支持 3 大系统平台，4 种编程语言应用，共涉及 200 多的实验场景，3000 多个实验参数，可以精细化的控制实验范围。混沌工程平台 chaosblade-box 支持实验工具托管，除已托管 chaosblade 外，还支持 Litmuschaos 实验工具。已登记使用企业 40 多家，其中已在工商银行、中国移动、小米、京东等企业中落地使用。

https://chaosblade.io/zh/docs/

做混沌实验、异常测试的，包含基础资源（CPU、内存、网络、磁盘、进程、内核、文件等）、多语言应用服务（Java、C++、NodeJS、Golang 等）、Kubernetes 平台（覆盖 Container、Pod、Node 资源场景，包含上述实验场景）。

安装

wget https://github.com/chaosblade-io/chaosblade/releases/download/v1.5.0/chaosblade-1.5.0-linux-amd64.tar.gz
tar -zxf chaosblade-1.5.0-linux-amd64.tar.gz
mv chaosblade-1.5.0 chaosblade

# 添加环境变量或添加软链，可以直接调用chaosblade
# 个人觉得加软链比较好看，若/usr/local/bin被加到环境变量的情况下，blade可以用
ln -s /data/zy/tools/chaosblade/blade /usr/bin/blade

# plan B, 如果只是在当前执行的话
export PATH=$PATH:/data/zy/tools/chaosblade
1
2
3
4
5
6
7
8
9
10

善用help

(base) darcyzhang@debian:~/tools$ blade help
An easy to use and powerful chaos engineering experiment toolkit

Usage:
  blade [command]

Available Commands:
  check       Check the environment for chaosblade
  create      Create a chaos engineering experiment，创建混沌实验
  destroy     Destroy a chaos experiment，销毁混沌实验
  help        Help about any command
  prepare     Prepare to experiment
  query       Query the parameter values required for chaos experiments
  revoke      Undo chaos engineering experiment preparation
  server      Server mode starts, exposes web services
  status      Query preparation stage or experiment status，查看实验状态
  version     Print version info

Flags:
  -d, --debug   Set client to DEBUG mode
  -h, --help    help for blade

Use "blade [command] --help" for more information about a command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

先来一个Demo

sudo blade create mem load --mem-percent 50
sudo blade status --type create --status success
1
2

# 创建占用50%内存的混沌实验	blade create help
(base) darcyzhang@debian:~$ sudo blade create mem load --mem-percent 50
{"code":200,"success":true,"result":"89b439c7a0b03625"}

# 查看实验状态	blade status help
(base) darcyzhang@debian:~$ sudo blade status --type create --status success
{
	"code": 200,
	"success": true,
	"result": [
		{
			"Uid": "acc4a866b5d1b4a1",
			"Command": "mem",
			"SubCommand": "load",
			"Flag": " --mem-percent=50",
			"Status": "Success",
			"Error": "",
			"CreateTime": "2022-05-15T16:03:26.234192228+08:00",
			"UpdateTime": "2022-05-15T16:03:26.677657158+08:00"
		}
	]
}

# 查看内存状态，这个内存占用需要一段时间才能创建完成
(base) darcyzhang@debian:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           49Gi       424Mi        24Gi        23Gi        24Gi        24Gi
Swap:         979Mi          0B       979Mi

# 删除混沌实验
(base) darcyzhang@debian:~$ sudo blade destroy acc4a866b5d1b4a1
{"code":200,"success":true,"result":{"target":"mem","action":"load","flags":{"mem-percent":"50"}}}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

创建常见实验

(base) darcyzhang@debian:~$ blade create --help
Create a chaos engineering experiment

Usage:
  blade create [command]

Aliases:
  create, c

Examples:
blade create cpu load --cpu-percent 60

Available Commands:
  cplus       C++ chaos experiments
  cpu         Cpu experiment，cpu实验
  cri         CRI experiment
  disk        Disk experiment，磁盘
  docker      Docker experiment，docker容器实验
  druid       Experiment with the Druid
  dubbo       Experiment with the Dubbo
  es          ElasticSearch experiment!
  file        File experiment
  gateway     gateway experiment!
  hbase       hbase experiment!
  http        http experiment
  jedis       jedis experiment
  jvm         Experiment with the JVM
  k8s         Kubernetes experiment
  kafka       kafka experiment
  lettuce     redis client lettuce experiment
  log         log experiment
  mem         Mem experiment，内存
  mongodb     MongoDB experiment
  mysql       mysql experiment
  network     Network experiment，网络
  process     Process experiment
  psql        Postgrelsql experiment
  rabbitmq    rabbitmq experiment
  redisson    redisson experiment
  rocketmq    Rocketmq experiment,can make message send or pull delay and exception
  script      Script chaos experiment
  servlet     java servlet experiment
  strace      strace experiment
  systemd     Systemd experiment
  tars        tars experiment

Flags:
  -a, --async             whether to create asynchronously, default is false
  -e, --endpoint string   the create result reporting address. It takes effect only when the async value is true and the value is not empty
  -h, --help              help for create
  -n, --nohup             used to internal async create, no need to config
      --uid string        Set Uid for the experiment, adapt to docker and cri

Global Flags:
  -d, --debug   Set client to DEBUG mode

Use "blade create [command] --help" for more information about a command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

timeout是创建混沌实验的一个常用参数，代表实验的持续时间，或者理解为实验多久后过期，创建实验最好带上这个参数，以防忘记销毁实验影响机器的正常使用。

CPU

(base) darcyzhang@debian:~$ blade create cpu --help
Cpu experiment, for example full load

Usage:
  blade create cpu [flags]
  blade create cpu [command]

Available Commands:
  fullload    cpu load		# cpu满载

Flags:
  -h, --help   help for cpu

Global Flags:
  -a, --async             whether to create asynchronously, default is false
  -d, --debug             Set client to DEBUG mode
  -e, --endpoint string   the create result reporting address. It takes effect only when the async value is true and the value is not empty
  -n, --nohup             used to internal async create, no need to config
      --uid string        Set Uid for the experiment, adapt to docker and cri

Use "blade create cpu [command] --help" for more information about a command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

创建CPU满载实验

blade create cpu load
1

内存

(base) darcyzhang@debian:~$ blade create mem --help
Mem experiment, for example load

Usage:
  blade create mem [flags]
  blade create mem [command]

Available Commands:
  load        mem load

Flags:
  -h, --help   help for mem

Global Flags:
  -a, --async             whether to create asynchronously, default is false
  -d, --debug             Set client to DEBUG mode
  -e, --endpoint string   the create result reporting address. It takes effect only when the async value is true and the value is not empty
  -n, --nohup             used to internal async create, no need to config
      --uid string        Set Uid for the experiment, adapt to docker and cri

Use "blade create mem [command] --help" for more information about a command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

blade create mem load --help

      --mem-percent string       内存占用百分比(0-100), 必须正整数
      --mode string              burn memory mode, cache or ram.
      --rate string              burn memory rate, unit is M/S, only support for ram mode.
      --reserve string           保留内存, unit is MB. 如果有 mem-percent, 优先 mem-percent.
      --timeout string           set timeout for experiment，实验超时时长
1
2
3
4
5
6
7

网络

blade create network --help

Available Commands:
  corrupt     Corrupt experiment
  delay       Delay experiment, 网络延时
  dns         Dns experiment
  drop        Drop experiment
  duplicate   Duplicate experiment
  loss        Loss network package，网络丢包
  occupy      Occupy the specify port，端口占用
  reorder     Reorder experiment
1
2
3
4
5
6
7
8
9
10
11

这里就不搞具体的例子了，一步一步的help下去，chaosblade的帮助文档还是写的很详细的。还有示例可以参考。

(base) darcyzhang@debian:~$ blade create network delay --help
Delay experiment

Usage:
  blade create network delay

Examples:	# 示例

# Access to native 8080 and 8081 ports is delayed by 3 seconds, and the delay time fluctuates by 1 second
blade create network delay --time 3000 --offset 1000 --interface eth0 --local-port 8080,8081

# Local access to external 14.215.177.39 machine (ping www.baidu.com obtained IP) port 80 delay of 3 seconds
blade create network delay --time 3000 --interface eth0 --remote-port 80 --destination-ip 14.215.177.39

# Do a 5 second delay for the entire network card eth0, excluding ports 22 and 8000 to 8080
blade create network delay --time 5000 --interface eth0 --exclude-port 22,8000-8080

Flags:
      --destination-ip string    目标IP，destination ip. Support for using mask to specify the ip range such as 92.168.1.0/24 or comma separated multiple ips, for example 10.0.0.1,11.0.0.1.
      --exclude-ip string        排除掉的ip，Exclude ips. Support for using mask to specify the ip range such as 92.168.1.0/24 or comma separated multiple ips, for example 10.0.0.1,11.0.0.1
      --exclude-port string      排除掉的端口，Exclude local ports. Support for configuring multiple ports, separated by commas or connector representing ranges, for example: 22,8000. This flag is invalid when --local-port or --remote-port is specified
      --force                    Forcibly overwrites the original rules
  -h, --help                     help for delay
      --ignore-peer-port         ignore excluding all ports communicating with this port, generally used when the ss command does not exist
      --install-path string      install path default /opt/chaosblade，use this flag when the channel is ssh
      --interface string         网口，Network interface, for example, eth0 (required)
      --local-port string        本地端口，Ports for local service. Support for configuring multiple ports, separated by commas or connector representing ranges, for example: 80,8000-8080
      --offset string            延时时间浮动时长，Delay offset time, ms
      --override-blade-release   Override blade release，use this flag when the channel is ssh
      --remote-port string       远程端口，Ports for remote service. Support for configuring multiple ports, separated by commas or connector representing ranges, for example: 80,8000-8080
      --time string              延时时间，Delay time, ms (required)
      --timeout string           实验超时时间，set timeout for experiment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

磁盘

blade create disk --help

Available Commands:
  burn        Increase disk read and write io load, 增加磁盘读写io
  fill        Fill the specified directory path，填充满指定目录
1
2
3
4
5

通过http远程创建实验

chaosblade提供通过http服务远程调用的方式，通过blade命令即可启动chaosblade的http服务。
在远程机器上启动chaosblade的驻守服务。

blade server start --port 8080
1

然后就可以本地机器远程调用了

# 创建
$ curl "http://192.168.1.106:8080/chaosblade?cmd=create%20cpu%20load%20--cpu-percent%2020"
{"code":200,"success":true,"result":"65161502e93cbe7e"}%
# 销毁
$ curl "http://192.168.1.106:8080/chaosblade?cmd=destroy%2065161502e93cbe7e"
{"code":200,"success":true,"result":{"action":"fullload","flags":{"cpu-percent":"20"},"target":"cpu"}}%
1
2
3
4
5
6

这里面的%20是空格，直接用空格不好使。

对容器的混沌实验

对容器的混沌实验也不是太复杂，只要加上docker关键字，指定容器id或容器名称即可

blade create docker network delay --interface eth0 --time 300 --container-name chttper
1

写的个小项目test一下

把项目通过docker部署起来

git clone https://gitee.com/bbjg001/httper
cd httper

# 构建项目镜像
docker build -t ihttper .
# 启动容器
docker run -tid --name chttper -p 8088:8088 ihttper 8088	# 前面的-p是宿主机和容器的端口映射，最后的端口是设置服务在容器内用的端口，前后要保持一致

# 测的接口是这样的
curl http://192.168.1.106:8088/objs
1
2
3
4
5
6
7
8
9
10

然后加上网络延时的混沌实验

blade create docker network delay --interface eth0 --time 300 --container-name chttper
1

用locust做一点子压力测试，网络延时前后测试结果如下。locust压力测试相关可以参见鄙人写的 Locust压力测试。

参考

混沌工程-阿里巴巴-chaosblade-安装与使用

chaosblade中文文档