k8s动态使用glusterfs存储_ks8 storageclass glasterfs

作者：菜鸟追梦旅行 | 2024-03-09 09:00:20

踩

ks8 storageclass glasterfs

k8s中运行Heketi服务

在使用kismatic安装k8s和glusterfs后，需要配置k8s默认动态管理（provisioning）使用glusterfs，以下内容是实现这样的目的。
Hekeli：glusterfs的restful端点，k8s向它发送命令来使用glusterfs。

Hekeli cli安装

下载Hekeli cli工具，安装在$PATH路径下
下载地址 https://github.com/heketi/heketi/releases/

Hekeli 服务端安装在k8s环境

在所有的glusterfs节点，创建hekeli的数据库存储目录、ssh免密码登录文件目录。

mkdir -p /data/heketi/{db,.ssh} && chmod 700 /data/heketi/.ssh1

生成ssh密钥

ssh-keygen -t rsa -b 2048 -f /data/heketi/.ssh/id_rsa

for NODE in node1 node2 node3; do scp -r /data/heketi/.ssh root@${NODE}:/data/heketi; done

for NODE in node1 node2 node3; do ssh-copy-id -i /data/heketi/.ssh/id_rsa.pub root@${NODE} ; done1
2
3
4
5

k8s运行heketi的服务
首先，标记能够运行heketi的节点

kubectl label node node1 storagenode=glusterfs1

运行heketi

kubectl apply -f heketi-secret.yaml -f heketi-deployment.json1

heketi-secret.yaml文件内容

apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  # base64 encoded password. E.g.: echo -n "password" | base64
  key: cGFzc3dvcmQ=
type: kubernetes.io/glusterfs1
2
3
4
5
6
7
8
9

heketi-deployment.json文件内容

{
  "kind": "List",
  "apiVersion": "v1",
  "items": [
    {
      "kind": "Service",
      "apiVersion": "v1",
      "metadata": {
        "name": "heketi",
        "labels": {
          "glusterfs": "heketi-service",
          "deploy-heketi": "support"
        },
        "annotations": {
          "description": "Exposes Heketi Service",
          "tags": "kubernetes,k8s,heketi",
          "traefik.backend.loadbalancer": "wrr",
          "traefik.backend.weight": "10",
          "traefik.enable": "true",
          "traefik.frontend.entryPoints": "http,https",
          "traefik.frontend.rule": "Host:heketi-api.example.com",
          "traefik.tags": "kubernetes"
        }
      },
      "spec": {
        "type": "NodePort",
        "selector": {
          "name": "heketi"
        },
        "ports": [
          {
            "name": "heketi",
            "port": 8080,
            "targetPort": 8080,
            “nodePort”: 30944
          }
        ]
      }
    },
    {
      "kind": "Deployment",
      "apiVersion": "apps/v1beta1",
      "metadata": {
        "name": "heketi",
        "labels": {
          "glusterfs": "heketi-deployment"
        },
        "annotations": {
          "description": "Defines how to deploy Heketi"
        }
      },
      "spec": {
        "replicas": 1,
        "strategy": {
          "rollingUpdate": {
            "maxSurge": 0,
            "maxUnavailable": 1
          },
          "type": "RollingUpdate"
        },
        "template": {
          "metadata": {
            "name": "heketi",
            "labels": {
              "name": "heketi",
              "glusterfs": "heketi-pod"
            }
          },
          "spec": {
            "terminationGracePeriodSeconds": 0,
            "nodeSelector": {
              "storagenode": "glusterfs"
            },
            "containers": [
              {
                "image": "heketi/heketi:5",
                "imagePullPolicy": "Always",
                "name": "heketi",
                "env": [
                  {
                    "name": "HEKETI_EXECUTOR",
                    "value": "ssh"
                  },
                  {
                    "name": "HEKETI_SSH_USER",
                    "value": "root"
                  },
                  {
                    "name": "HEKETI_SSH_PORT",
                    "value": "22"
                  },
                  {
                    "name": "HEKETI_SSH_KEYFILE",
                    "value": "/root/.ssh/id_rsa"
                  },
                  {
                    "name": "HEKETI_ADMIN_KEY",
                    "valueFrom": {
                      "secretKeyRef": {
                        "name": "heketi-secret",
                        "key": "key"
                      }
                    }
                  }
                ],
                "ports": [
                  {
                    "containerPort": 8080
                  }
                ],
                "volumeMounts": [
                  {
                    "name": "heketi-ssh-key",
                    "mountPath": "/root/.ssh"
                  },
                  {
                    "name": "heketi-db",
                    "mountPath": "/var/lib/heketi"
                  }
                ],
                "readinessProbe": {
                  "timeoutSeconds": 3,
                  "initialDelaySeconds": 3,
                  "httpGet": {
                    "path": "/hello",
                    "port": 8080
                  }
                },
                "livenessProbe": {
                  "timeoutSeconds": 3,
                  "initialDelaySeconds": 15,
                  "httpGet": {
                    "path": "/hello",
                    "port": 8080
                  }
                }
              }
            ],
            "volumes": [
              {
                "name": "heketi-ssh-key",
                "hostPath": {
                  "path": "/data/heketi/.ssh"
                }
              },
              {
                "name": "heketi-db",
                "hostPath": {
                  "path": "/data/heketi/db"
                }
              }
            ]
          }
        }
      }
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159

这里配置Heketi的service在NodeIP：30944上
通过命令检查heketi服务

curl -s http://nodeIP:30944/hello1

通过cli连接

heketi-cli --user admin --secret password --server http://node1:30944 cluster list1

导入glusterfs集群拓扑（topology）信息

heketi-cli --user admin --secret password --server http://node1:30944 topology load --json heketi-topology.json1

heketi-topology.json的文件内容为

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [ "192.168.122.112" ],
              "storage": [ "192.168.122.112" ] },
            "zone": 1
          },
          "devices": [
            "/dev/vdb"
          ]
        }
      ]
    }
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

有关glusterfs集群的topology的配置参考
https://github.com/heketi/heketi/blob/master/docs/admin/topology.md
使用以下命令获得cluster id

heketi-cli --user admin --secret password --server http://192.168.122.112:30944 cluster list1

k8s使用glusterfs

k8s上定义glusterds的default storageclass，然后通过pvc来动态使用glusterfs存储

kubectl apply -f heketi-storageclass.yaml1

heketi-storageclass.yaml的内容如下

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://192.168.122.112:30944"
  clusterid: "e4b83af0a6260975ed1001ea3e89a0ad"
  restuser: "admin"
  secretNamespace: "default"
  secretName: "heketi-secret"
  volumetype: "none"1
2
3
4
5
6
7
8
9
10
11
12
13
14

kubectl apply -f heketi-pvc.yaml
kubectl get pvc test-claim
kubectl get pv1
2
3

heketi-pvc.yaml的文件内容

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi1
2
3
4
5
6
7
8
9
10

使用命令查看glusterfs中的volume

heketi-cli --user admin --secret password --server http://192.168.122.112:30944 volume list1

创建pod使用pvc

kubectl apply -f nginx-pod.yaml1

nginx-pod.yaml的内容

kind: Pod
apiVersion: v1
metadata:
  name: nginx-with-pv
spec:
  containers:
    - name: frontend
      image: nginx:stable-alpine
      volumeMounts:
      - mountPath: "/usr/share/nginx/html"
        name: pv
  nodeSelector:
    storagenode: "glusterfs"
  volumes:
    - name: pv
      persistentVolumeClaim:
        claimName: test-claim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/菜鸟追梦旅行/article/detail/214397