【k8s中安装rabbitmq】k8s中基于安装rabbitmq并搭建镜像集群-pvc版

简介

该文搭建的rabbitmq集群是采用rabbitmq_peer_discovery_k8s的形式进行搭建，是通过该插件自动从k8s api中读取节点信息并组建rabbitmq集群，采用搭建方式是一个statefulset,三个副本的形式，所以要保证数据的持久可以采用hostpath+节点亲和性，也可以采用pvc的形式，本文将讲述采用pvc的形式来报障数据的持久化。

一.条件及环境说明

k8s版本k8s-1.29.4，采用共享存储的形式搭建，共享存储采用自建的root-ceph的形式，如果是采用阿里云的集群可以通过挂载硬盘、挂载nas或者对象存储来实现数据持久化，采用共享存储的缺点就是普遍存在读写性能较差的问题。

4.2.创建configmap配置

将如下配置保存到autotest-rabbitmq-config.yaml，配置中配置默认的vhost和用户密码，并初始化好集群节点【安装前规划好节点名称】信息。

apiVersion: v1
kind: ConfigMap
metadata:
  name: autozx-rabbitmq-config
  namespace: zx-app
  labels:
    appname: pcauto-zx
    app: autozx-rabbitmq-config 
data:
  enabled_plugins: |
    [rabbitmq_management,rabbitmq_peer_discovery_k8s].
  rabbitmq.conf: |
    cluster_name = autozx-rabbitmq
    listeners.tcp.default = 5672
    
    default_vhost = /
    default_user = admin
    default_pass = pconline

    default_user_tags.administrator = true
    default_user_tags.management = true
    default_user_tags.custom_tag = true

    channel_max = 1024
    tcp_listen_options.backlog = 2048

    cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.node_cleanup.interval = 30
    cluster_formation.node_cleanup.only_log_warning = true

    cluster_partition_handling = autoheal

    queue_master_locator=min-masters
    loopback_users.guest = false
    cluster_formation.k8s.hostname_suffix = .autozx-rabbitmq.zx-app.svc.cluster.local
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

4.3.创建statefulset和service headless配置

采用3副本

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autozx-rabbitmq 
  namespace: zx-app 
  labels:
    appname: pcauto-zx
    app: autozx-rabbitmq
spec:
  serviceName: "autozx-rabbitmq"
  replicas: 3
  selector:
    matchLabels:
      app: autozx-rabbitmq
  template:
    metadata:
      labels:
        app: autozx-rabbitmq
    spec:
      containers:
      - name: rabbitmq-server
        image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management 
        imagePullPolicy: IfNotPresent
        env:
        - name: RABBITMQ_ERLANG_COOKIE
          value: "YZSDHWMFSMKEMBDHSGGZ"
        - name: K8S_SERVICE_NAME
          value: autozx-rabbitmq 
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
        ports:
        - name: http
          containerPort: 15672 
        - name: amqp
          containerPort: 5672 
        readinessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        volumeMounts:
        - name: rbmq-data
          mountPath: /var/lib/rabbitmq
        - name: rabbitmq-config-volume
          mountPath: /etc/rabbitmq/ 
      restartPolicy: Always 
      serviceAccountName: rabbitmq-cluster
      terminationGracePeriodSeconds: 30
      volumes:
        - name: rabbitmq-config-volume
          configMap:
            name: autozx-rabbitmq-config
  volumeClaimTemplates:
  - metadata:
      name: rbmq-data
    spec:
      accessModes:
      - ReadWriteMany
      storageClassName: example-storageclass
      resources:
        requests:
          storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: autozx-rabbitmq
  namespace: zx-app
  labels:
    appname: pcauto-zx
    app: autozx-rabbitmq
spec:
  ports:
  - port: 5672
  clusterIP: None
  selector:
    app: autozx-rabbitmq
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

注：如果为了避免相同的pod分配到同一节点可以加上如下配置：

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - autozx-rabbitmq
              topologyKey: "kubernetes.io/hostname"
1
2
3
4
5
6
7
8
9
10

4.4.授权配置

配置serviceAccount、role、RoleBinding来给statefulset进行授权读取节点信息

apiVersion: v1
kind: Service
metadata:
  name: autozx-rabbitmq-manage
  namespace: zx-app
  labels:
    app: autozx-rabbitmq-manage
    appname: pcauto-zx
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 15672
    name: http
  selector:
    app: autozx-rabbitmq
  type: LoadBalancer 
[root@autobbs-docker-240-213 rabbitmq]# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq-cluster
  namespace: zx-app
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq-cluster
  namespace: zx-app
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq-cluster
  namespace: zx-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rabbitmq-cluster
subjects:
- kind: ServiceAccount
  name: rabbitmq-cluster
  namespace: zx-app
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

4.5.创建service配置

给amqp端口5672和管理端口15672配置一个service的代理服务.

apiVersion: v1
kind: Service
metadata:
  name: autozx-rabbitmq-manage
  namespace: zx-app
  labels:
    app: autozx-rabbitmq-manage
    appname: pcauto-zx
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 15672
    name: http
  selector:
    app: autozx-rabbitmq
  type: LoadBalancer 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

五.安装完后的配置

将rabbitmq的集群设置成3节点的镜像集群，在4.4步骤操作完以后就可以通过loadbalancer的IP进行登录控制台了：http://ip:15672,用configmap中设置的 default_user和default_pass来登录。

镜像模式设置：

设置后：

镜像模式可以通过命令设置：
设置demo的vhost的镜像执行命令：
rabbitmqctl set_policy -p demo ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
设置默认vhost /的镜像模式：
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

六.安装说明

• 使用配置前将配置中的autozx替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-zx，修改成自己的应用名称，该标签也可以删除。
• 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的rabbitmq:3.12.14-management 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用rabbitmq:3.12.14-management 。否则配置一个可以通的镜像地址。
• 默认的用户名和密码是admin。
• pvc模板中的存储类：example-storageclass 修改成自己的存储类
• "五.安装后的配置"由于配置方法都是一样，就采用了上一篇文章中的截图