热门标签
热门文章
- 1jenkins自动化部署_jenkins打包,发布,部署
- 2opencv图像美化_opencv 图像美化
- 3计算机基础知识和实践技能300分,2019年河北省普通高职单招考试十类和对口电子电工类、计算机类联考考试内容...
- 4VideoCrafter - 文本生成视频
- 5openMP编程详解（囊括所有基本指令）
- 6函授大专计算机考证_函授大专计算机专业可以考什么证
- 7一分钟教你把Notion变为中文版_notion怎么变成中文版
- 8axios详解以及完整封装方法_axios封装
- 9esxi6.7安装MacOS 10.15_esxi6.7安装mac os 卡住
- 10Android HAL层添加HIDL实例实现串口通信_android 新增hidl接口
当前位置: article > 正文
使用 BPF 监控 Kubernetes 集群(k8s BPF 工具 kubectl-trace认知
作者：盐析白兔 | 2024-03-23 15:26:31
赞
踩
使用 BPF 监控 Kubernetes 集群(k8s BPF 工具 kubectl-trace认知
写在前面
- 学习中遇到，整理分享，博文内容涉及：
kubectl-trace安装，在节点，容器中如何使用- 需要注意的问题：
job闪完成，一直Pending状态解决 - 理解不足小伙伴帮忙指正
不必太纠结于当下，也不必太忧虑未来，当你经历过一些事情的时候，眼前的风景已经和从前不一样了。——村上春树
kubectl-trace 安装
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$curl -L -o kubectl-trace.tar.gz https://github.com/iovisor/kubectl-trace/releases/download/v0.1.0-rc.1/kubectl-trace_0.1.0-rc.1_linux_amd64.tar.gz
└─$pwd
/root/ansible/trace
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$ls
kubectl-trace.tar.gz
- 1
- 2
- 3
- 4
- 5
- 6
- 7
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$tar -xvf kubectl-trace.tar.gz
LICENSE
README.md
kubectl-trace
trace-runner
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$mv kubectl-trace /usr/local/bin/kubectl-trace
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
查看版本，检查安装是否成功
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl trace version
git commit: d34d1d586b110af718aeadc9f3213c78e543a961
build date: 2019-09-19 23:00:13 +0800 CST
- 1
- 2
- 3
- 4
如何使用
这里我们看一下帮助文档
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl trace run --help Execute a bpftrace program on resources Usage: trace run (POD | TYPE/NAME) [-c CONTAINER] [--attach] [flags] Examples: # Count system calls using tracepoints on a specific node kubectl trace run node/kubernetes-node-emt8.c.myproject.internal -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }' # Execute a bpftrace program from file on a specific node kubectl trace run node/kubernetes-node-emt8.c.myproject.internal -f read.bt # Run an bpftrace inline program on a pod container kubectl trace run pod/nginx -c nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" # Run a bpftrace inline program on a pod container with a custom image for the init container responsible to fetch linux headers kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); } --init-imagename=quay.io/custom-init-image-name --fetch-headers" # Run a bpftrace inline program on a pod container with a custom image for the bpftrace container that will run your program in the cluster kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); } --imagename=quay.io/custom-bpftrace-image-name" ............... ┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
在节点上使用
在使用之前，可能需要做一些准备工作，提前拉去镜像
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible k8s_node -m shell -a "docker pull quay.io/iovisor/kubectl-trace-bpftrace:latest" -i host.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible k8s_node -m shell -a "docker pull quay.io/iovisor/kubectl-trace-init:latest" -i host.yaml
- 1
- 2
- 3
- 4
使用 tracepoints 在特定节点上计算系统调用次数(帮助文档中的Demo,这里实际上是动态跟踪)：
kubectl trace run node/kubernetes-node-emt8.c.myproject.internal -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }'
- 1
kprobe:do_sys_open 是一个 bpftrace 的探针（probe），用于跟踪 Linux 内核中的 do_sys_open 系统调用。
实际上kprobe 是一种用于在内核函数入口处执行的探针类型,属于动态跟踪。do_sys_open 是一个内核函数，它在打开文件时被调用。而 tracepoint 属于静态跟踪
实际的 动态跟踪 Demo
运行方式，这里使用编码方式，传递字符串
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl trace run node/vms103.liruilongs.github.io -e 'kprobe:do_sys_open { printf("%s: %s\n", comm, str(arg1)) }'
trace a659c18c-e50c-11ee-ba0d-000c290e5d5f created
- 1
- 2
- 3
会启动一个 job 在对应的 节点上
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get jobs
NAME COMPLETIONS DURATION AGE
kubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f 0/1 20s 20s
webhook-cert-setup 0/1 124d 124d
- 1
- 2
- 3
- 4
- 5
获取 pod 信息
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get jobs | grep a659c18c-e50c-11ee-ba0d-000c290e5d5f
kubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f 0/1 29s 29s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get pods | grep a659c18c-e50c-11ee-ba0d-000c290e5d5f
kubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f-7292n 1/1 Running 0 65s
- 1
- 2
- 3
- 4
- 5
- 6
查看 Job 日志，即上面的 跟踪日志
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl logs kubectl-trace-a659c18c-e50c-11ee-ba0d-000c290e5d5f-7292n --tail=3
dockerd: /docker/data/overlay2/b212e7a41dd9417f549f234ce768a3fd0fc0e0baf
dockerd: /docker/data/image/overlay2/imagedb/content/sha256/1f0ce7a730e7
dockerd: /docker/data/image/overlay2/imagedb/metadata/sha256/1f0ce7a730e
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
静态跟踪 Demo
这里的静态跟踪实际上是帮助文档中讲的 tracepoint 内核跟踪点，需要注意部分机器可能需要添加 --fetch-headers
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl trace run vms105.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }" --fetch-headers
trace 305a7d60-e5a0-11ee-ba88-000c290e5d5f created
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods -w
NAME READY STATUS RESTARTS AGE
hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18d
kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 0/1 Init:0/1 0 4s
kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 0/1 PodInitializing 0 15s
kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 1/1 Running 0 21s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
查看日志信息
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl logs kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt
Defaulted container "kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f" out of: kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f, kubectl-trace-init (init)
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times
Attaching 1 probe...
- 1
- 2
- 3
- 4
- 5
在跟踪节点传递结束信息（重新打个页面执行）
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.105 -m shell -a "pgrep bpftrace" -i host.yaml
192.168.26.105 | CHANGED | rc=0 >>
37312
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible 192.168.26.105 -m shell -a "kill -2 37312" -i host.yaml
192.168.26.105 | CHANGED | rc=0 >>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
对应的跟踪日志信息输出
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl logs kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f-ctfl6 -f Defaulted container "kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f" out of: kubectl-trace-305a7d60-e5a0-11ee-ba88-000c290e5d5f, kubectl-trace-init (init) if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times Attaching 1 probe... @[bash]: 1 @[systemd-udevd]: 1 @[calico]: 2 @[containerd]: 6 @[nsenter]: 15 @[kubelet]: 18 @[kube-proxy]: 34 @[runc:[2:INIT]]: 207 @[exe]: 207 @[runc]: 208 @[containerd-shim]: 220 @[calico-node]: 303 @[cri-dockerd]: 313 @[dockerd]: 3907
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
pod 状态为完成
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Completed 0 31m
- 1
- 2
- 3
- 4
- 5
运行方式通过 文件的方式，使用 bpftrace 自带的脚步执行
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl trace run vms105.liruilongs.github.io -f opensnoop.bt --fetch-headers trace e0b7df1e-e5a1-11ee-898f-000c290e5d5f created ┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl get pods | grep e0b7df1e-e5a1-11ee-898f-000c290e5d5f kubectl-trace-e0b7df1e-e5a1-11ee-898f-000c290e5d5f-wv8w2 0/1 PodInitializing 0 12s ┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl trace logs e0b7df1e-e5a1-11ee-898f-000c290e5d5f if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times Attaching 6 probes... Tracing open syscalls... Hit Ctrl-C to end. PID COMM FD ERR PATH 1038 dockerd 133 0 /docker/data/image/overlay2/imagedb/content/sha256/4873874c08ef 1038 dockerd 133 0 /docker/data/image/overlay2/imagedb/content/sha256/4873874c08ef 1038 dockerd -1 2 /docker/data/image/overlay2/imagedb/metadata/sha256/4873874c08e 1316 kubelet 17 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.stat 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.usage 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.max_u 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.failc 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.limit 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw 1316 kubelet 19 0 /sys/fs/cgroup/memory/system.slice/kubelet.service/memory.memsw ............................
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
bpftrace 自带了opensnoop.bt，这个工具可以同时对每个系统调用的开始和结束位置进行跟踪，然后将结果分列输出:
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$cat opensnoop.bt #!/usr/bin/bpftrace /* * opensnoop Trace open() syscalls. * For Linux, uses bpftrace and eBPF. * * Also a basic example of bpftrace. * * USAGE: opensnoop.bt * * This is a bpftrace version of the bcc tool of the same name. * * Copyright 2018 Netflix, Inc. * Licensed under the Apache License, Version 2.0 (the "License") * * 08-Sep-2018 Brendan Gregg Created this. */ BEGIN { printf("Tracing open syscalls... Hit Ctrl-C to end.\n"); printf("%-6s %-16s %4s %3s %s\n", "PID", "COMM", "FD", "ERR", "PATH"); } tracepoint:syscalls:sys_enter_open, tracepoint:syscalls:sys_enter_openat { @filename[tid] = args->filename; } tracepoint:syscalls:sys_exit_open, tracepoint:syscalls:sys_exit_openat /@filename[tid]/ { $ret = args->ret; $fd = $ret >= 0 ? $ret : -1; $errno = $ret >= 0 ? 0 : - $ret; printf("%-6d %-16s %4d %3d %s\n", pid, comm, $fd, $errno, str(@filename[tid])); delete(@filename[tid]); } END { clear(@filename); }
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
容器中使用
容器相关的这里不做演示，下面是帮助文档的 Demo
在 Pod 容器上运行内联的 bpftrace 程序：
kubectl trace run pod/nginx -c nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"
- 1
在 Pod 容器上使用自定义镜像运行内联的 bpftrace 程序：
kubectl trace run pod/nginx nginx -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }" --imagename=quay.io/custom-bpftrace-image-name
- 1
需要注意的问题
job 闪完成
在编译或构建过程中，表示缺少了必要的头文件,job 启 Pod 之后会马上闪退
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl trace run node/vms103.liruilongs.github.io -f opensnoop.bt
trace fc86c785-e513-11ee-8e04-000c290e5d5f created
- 1
- 2
- 3
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl get pods | grep fc86c785-e513-11ee-8e04-000c290e5d5f kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 ContainerCreating 0 8s ┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl get pods kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm -w NAME READY STATUS RESTARTS AGE kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Completed 0 19s kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Completed 0 20s kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Completed 0 21s kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Terminating 0 26s kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm 0/1 Terminating 0 26s ^C┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$ ┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace] └─$kubectl logs kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm Error from server (NotFound): pods "kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm" not found
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
日志信息
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two ││ times │
│/bpftrace/include/clang_workarounds.h:14:10: fatal error: 'linux/types.h' file not found ││exit status 1
- 1
- 2
执行环境
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$hostnamectl
Static hostname: vms100.liruilongs.github.io
Icon name: computer-vm
Chassis: vm
Machine ID: e93ae3f6cb354f3ba509eeb73568087e
Boot ID: 51ad5f1933914654affd2dcf9ebca862
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 5.4.266-1.el7.elrepo.x86_64
Architecture: x86-64
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
解决办法： 添加 --fetch-headers 尝试 https://github.com/iovisor/kubectl-trace/issues/177
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl trace run vms105.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }" --fetch-headers trace 3d9981a0-e59b-11ee-a570-000c290e5d5f created ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get pods NAME READY STATUS RESTARTS AGE hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18d kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 12s ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get pods -w NAME READY STATUS RESTARTS AGE hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18d kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 33s kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 41s ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get pods -w NAME READY STATUS RESTARTS AGE hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18d kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 0/1 Init:0/1 0 64s ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get pods NAME READY STATUS RESTARTS AGE hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d16h ago) 18d kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt 1/1 Running 0 16m
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
一直 Pending 状态
这里需要注意，如果是 master 节点做分析，需要把对应的 master 的污点去掉，负责无法调度
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl trace run vms102.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }"
trace 4f30d986-e595-11ee-a5ef-000c290e5d5f created
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-webhook-deployment-7f599b95c4-hjx86 1/1 Running 1 (2d15h ago) 18d
kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f-xw6qm 0/1 Pending 0 7s
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl describe pods kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f-xw6qm
..................
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 23s default-scheduler 0/6 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl trace delete 4f30d986-e595-11ee-a5ef-000c290e5d5f
trace job kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f deleted
trace configuration kubectl-trace-4f30d986-e595-11ee-a5ef-000c290e5d5f deleted
- 1
- 2
- 3
- 4
在实际生产中可以考虑通过 sidecar 边车的方式运行，监视 pod 中的所有容器。这里不做分享，感兴趣小伙伴可以研究下 _
博文部分内容参考
© 文中涉及参考链接内容版权归原作者所有，如有侵权请告知
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/盐析白兔/article/detail/296056
推荐阅读
相关标签
Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。
