当前位置:   article > 正文

通过minio operator在k8s中部署minio tenant集群

minio operator

1、概述

MinIO是kubernetes原生的高性能对象存储,兼容Amazon的s3 API。

MinIO Operator是一个工具,该工具扩展了k8s的api,可以通过minio operator在公有云和私有云上来部署MinIO Tenants

本文就通过一步一步的操作,来演示如何在一个现存的k8s集群中部署一个minio集群。

2、MinIO集群部署过程

2.1、部署minio operator

本文档中,我们介绍的都是通过minio operator这个工具来部署minio的租户集群,那么首先,我们来安装minio operator

2.1.1、安装kubectl-minio插件

通过以下的命令安装minio operaor和minio插件(kubectl-minio)

wget http://xxxxx/kubernetes/kubectl-plugin/kubectl-minio_linux_amd64.zip
unzip kubectl-minio_linux_amd64.zip "kubectl-minio" -d /usr/local/bin/

kubectl minio version
  • 1
  • 2
  • 3
  • 4

如果出现以下的部分,说明kubectl-minio插件已经安装成功

[root@nccztsjb-node-23 ~]# kubectl minio version
v4.4.16
  • 1
  • 2

注意:这里的kubectl-minio插件压缩包已经下载到本地,也可以通过github上去获取需要的版本。

地址:https://github.com/minio/operator/releases

在这里插入图片描述

建议:直接下载zip包,下载二进制包,如果kubectl minio不是具体的版本,而是类似如下的信息:

[rootb@125 ~]# kubectl minio version
DEVELOPMENT.GOGET
[root@b125 ~]# 
  • 1
  • 2
  • 3

可能会导致后续的初始化和安装失败,并且operator打开之后,功能会不完整。

2.1.2、初始化minio operator

kubectl-minio插件完成之后,需要进行初始化的操作,即通过kubectl minio命令安装minio operator服务

a、执行以下的命令进行初始化

kubectl minio init \
  --image=1xxxx/minio/operator:v4.4.16 \
  --console-image=xxxx/minio/console:v0.15.13
  • 1
  • 2
  • 3

注意:可以提前将镜像下载到本地的harbor,因为,本地有这样的镜像,在不同的环境部署的时候会节省时间。如果环境可以联网,那么是可以不加–image和–console-image参数的部分

安装过程

[root@nccztsjb-node-23 ~]# kubectl minio init \
>   --image=xxxx/minio/operator:v4.4.16 \
>   --console-image=1xxxx/minio/console:v0.15.13
namespace/minio-operator created
serviceaccount/minio-operator created
clusterrole.rbac.authorization.k8s.io/minio-operator-role created
clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created
customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created
service/operator created
deployment.apps/minio-operator created
serviceaccount/console-sa created
clusterrole.rbac.authorization.k8s.io/console-sa-role created
clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding created
configmap/console-env created
service/console created
deployment.apps/console created
-----------------

To open Operator UI, start a port forward using this command:

kubectl minio proxy -n minio-operator 

-----------------
[root@nccztsjb-node-23 ~]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24

b、查看minio operator会被安装在minio-operator命名空间里

[root@nccztsjb-node-23 ~]# kubectl get all -n minio-operator 
NAME                                 READY   STATUS    RESTARTS   AGE
pod/console-7796fcb6c4-q8tf4         1/1     Running   0          64s
pod/minio-operator-5d95d44c5-lffkr   1/1     Running   0          64s
pod/minio-operator-5d95d44c5-q7zdv   1/1     Running   0          64s

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/console    ClusterIP   10.105.37.171   <none>        9090/TCP,9443/TCP   64s
service/operator   ClusterIP   10.100.4.192    <none>        4222/TCP,4221/TCP   64s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/console          1/1     1            1           64s
deployment.apps/minio-operator   2/2     2            2           64s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/console-7796fcb6c4         1         1         1       64s
replicaset.apps/minio-operator-5d95d44c5   2         2         2       64s
[root@nccztsjb-node-23 ~]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18

当所有的pod都为Running状态的时候,表示服务已经ready了。

c、登录operator控制台

kubectl minio proxy -n minio-operator 
  • 1

操作日志

[root@nccztsjb-node-23 ~]# kubectl minio proxy -n minio-operator
Starting port forward of the Console UI.

To connect open a browser and go to http://localhost:9090

Current JWT to login: eyJhbGciOiJxxxxxx

Forwarding from 0.0.0.0:9090 -> 9090
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8

浏览器打开该地址(执行命令的主机):9090

在这里插入图片描述

输入JWT(上面执行命令的时候显示的 xxxxxxxx)

在这里插入图片描述

这样就登录了这个控制台了,当然我们可以在这里通过界面的操作来创建minio集群,但是本文档我们主要是通过命令行的方式来进行部署,这个后面会讲到。

2.2、部署minio tenant集群

我们在上文中已经部署了minio operator和对应的插件工具kubectl minio,在这部分我们要进行tenant集群的部署。

2.2.1、创建StorageClass

我们会通过storageclass和具体的pv进行绑定,来做为minio本地的持久化存储,当pv创建好之后,创建minio tenant集群的时候,会自动创建pvc然后通过这个storageclass和本地的pv进行绑定

通过以下的命令来创建storageclass

kubectl apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: minio-local-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
EOF
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9

注意:volumeBindingMode: WaitForFirstConsumer必须要设置为WaitForFirstConsumer,这样就会在pvc和pv进行绑定的时候,根据消费的情况进行绑定,而不是随机的绑定。

随机的绑定有可能导致,一个pod关联了不同主机的pv导致启动失败。

[root@nccztsjb-node-23 ~]# kubectl get sc
NAME                  PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
minio-local-storage   kubernetes.io/no-provisioner   Retain          WaitForFirstConsumer   false                  28h
[root@nccztsjb-node-23 ~]# 
  • 1
  • 2
  • 3
  • 4
2.2.2、创建pv

对minio来说,使用local类型的pv进行数据的持久化的存储。

在每个节点创建不同的目录,用于local类型的本地存储目录

mkdir -p /data/minio/{pv1,pv2,pv3,pv4}
  • 1

这里3个节点,一共12个volume,每个节点4个pv

通过以下的命令创建local类型的pv

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-23-01
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-23
              
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-23-02
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-23
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-23-03
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-23
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-23-04
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-23   
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-24-01
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-24
              
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-24-02
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-24
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-24-03
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-24
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-24-04
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-24  
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-25-01
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv1
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-25
              
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-25-02
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv2
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-25
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-25-03
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv3
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-25
---
apiVersion: v1
kind: PersistentVolume
metadata:
   name: nccztsjb-node-25-04
spec:
   capacity:
      storage: 10Gi
   volumeMode: Filesystem
   accessModes:
   - ReadWriteOnce
   persistentVolumeReclaimPolicy: Retain
   storageClassName: minio-local-storage
   local:
      path: /data/minio/pv4
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - nccztsjb-node-25
EOF
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198
  • 199
  • 200
  • 201
  • 202
  • 203
  • 204
  • 205
  • 206
  • 207
  • 208
  • 209
  • 210
  • 211
  • 212
  • 213
  • 214
  • 215
  • 216
  • 217
  • 218
  • 219
  • 220
  • 221
  • 222
  • 223
  • 224
  • 225
  • 226
  • 227
  • 228
  • 229
  • 230
  • 231
  • 232
  • 233
  • 234
  • 235
  • 236
  • 237
  • 238
  • 239
  • 240
  • 241
  • 242
  • 243
  • 244
  • 245
  • 246
  • 247
  • 248
  • 249
  • 250
  • 251
  • 252
  • 253
  • 254
  • 255
  • 256
  • 257
  • 258
  • 259
  • 260
  • 261
  • 262
  • 263
  • 264
  • 265
  • 266
  • 267
  • 268
  • 269
  • 270
  • 271
  • 272
  • 273
  • 274
  • 275
  • 276
  • 277
  • 278
  • 279
  • 280

每个pv都是使用local类型,节点亲和的方式和具体的节点进行绑定

[root@nccztsjb-node-23 ~]# kubectl get pv
NAME                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS          REASON   AGE
nccztsjb-node-23-01   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-23-02   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-23-03   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-23-04   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-24-01   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-24-02   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-24-03   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-24-04   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-25-01   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-25-02   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-25-03   10Gi       RWO            Retain           Available           minio-local-storage            3s
nccztsjb-node-25-04   10Gi       RWO            Retain           Available           minio-local-storage            3s
[root@nccztsjb-node-23 ~]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15

这样,我们就创建好了,这12个pv.

注意、注意、注意:storageClassName: minio-local-storage,必须使用第一个步骤中的storageclass,否则在创建集群的时候,pv和pvc的绑定就会有问题!

2.2.3、创建命名空间

需要将minio租户放在一个特定的命名空间建中

kubectl create ns minio-tenant-1
  • 1
2.2.4、创建minio tenant 集群

本地化存储已经具备,下面就来创建minio租户集群

kubectl minio tenant create minio-tenant-1  \
  --servers 3 \
  --volumes 12 \
  --capacity 120Gi \
  --storage-class  minio-local-storage \
  --namespace minio-tenant-1 \
  --image xxxxxx/minio/minio:RELEASE.2022-04-16T04-26-02Z
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7

这里我们的集群有3个节点,–servers 3,12个卷,每个pv 10g,就是120Gi的容器(capacity),注意,这里的storageclass这要指定minio-local-storage,另外,如果环境可以联网,也不需要指定–image操作

操作过程

[root@nccztsjb-node-23 ~]# kubectl minio tenant create minio-tenant-1  \
>   --servers 3 \
>   --volumes 12 \
>   --capacity 120Gi \
>   --storage-class  minio-local-storage \
>   --namespace minio-tenant-1 \
>   --image xxxx/minio/minio:RELEASE.2022-04-16T04-26-02Z

Tenant 'minio-tenant-1' created in 'minio-tenant-1' Namespace

  Username: admin 
  Password: xxxxxxx-02a3-4366-a2a4-5b74cb26a185 
  Note: Copy the credentials to a secure location. MinIO will not display these again.

+-------------+------------------------+----------------+--------------+--------------+
| APPLICATION | SERVICE NAME           | NAMESPACE      | SERVICE TYPE | SERVICE PORT |
+-------------+------------------------+----------------+--------------+--------------+
| MinIO       | minio                  | minio-tenant-1 | ClusterIP    | 443          |
| Console     | minio-tenant-1-console | minio-tenant-1 | ClusterIP    | 9443         |
+-------------+------------------------+----------------+--------------+--------------+

[root@nccztsjb-node-23 ~]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22

这里面会给一个minio的管理用户密码,需要妥善保存。

查看pod的状态

[root@nccztsjb-node-23 ~]# kubectl get all -n minio-tenant-1 
NAME                        READY   STATUS    RESTARTS      AGE
pod/minio-tenant-1-ss-0-0   1/1     Running   0             43s
pod/minio-tenant-1-ss-0-1   1/1     Running   1 (35s ago)   43s
pod/minio-tenant-1-ss-0-2   1/1     Running   0             42s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/minio                    ClusterIP   10.104.251.7   <none>        443/TCP    43s
service/minio-tenant-1-console   ClusterIP   10.102.87.98   <none>        9443/TCP   43s
service/minio-tenant-1-hl        ClusterIP   None           <none>        9000/TCP   43s

NAME                                   READY   AGE
statefulset.apps/minio-tenant-1-ss-0   3/3     43s
[root@nccztsjb-node-23 ~]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

pod都为Running的状态。

并且查看operator中的集群状态信息

在这里插入图片描述

当为health的时候,说明集群已经初始化了,没有问题。

在这里插入图片描述

OK,到这里,就完成了通过minio operator来安装minio集群的过程。

通过minio operator可以做很多的对集群的管理的操作。

3、kubectl minio常见操作

3.1、查看当前的tenant

[root@nccztsjb-node-23 ~]# kubectl minio tenant list

Tenant 'minio-tenant-1', Namespace 'minio-tenant-1', Total capacity 120 GiB

  Current status: Initialized 
  MinIO version: 172.20.58.152/minio/minio:RELEASE.2022-04-16T04-26-02Z 

[root@nccztsjb-node-23 ~]#
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8

3.2、删除operator

kubectl minio delete
  • 1

3.3、删除tenant

kubectl minio tenant delete minio-tenant-1 --namespace minio-tenant-1
  • 1

3.4、删除pvc、pv

kubectl delete pvc --all -n minio-tenant-1 
kubectl delete pv --all -n minio-tenant-1 
  • 1
  • 2

3.5、删除命名空间

kubectl delete ns minio-tenant-1
  • 1

4、问题汇总

在部署的过程中,启动tenant集群无法启动,始终报错如下:

API: SYSTEM()
Time: 07:34:57 UTC 04/24/2022
Error: Detected unexpected disk ordering refusing to use the disk - poolID: 1st, found disk mounted at (set=1st, disk=3rd) expected mount at (set=1st, disk=9th): https://minio-tenant-1-pool-0-2.minio-tenant-1-hl.minio-tenant-1.svc.cluster.local:9000/export2(45e46618-72a1-41a4-99f5-91deec0b7208) (*errors.errorString)
       1: cmd/erasure-sets.go:451:cmd.newErasureSets.func1.1()

API: SYSTEM()
Time: 07:34:57 UTC 04/24/2022
Error: Detected unexpected disk ordering refusing to use the disk - poolID: 1st, found disk mounted at (set=1st, disk=1st) expected mount at (set=1st, disk=7th): /export2(583c2c4a-05a5-4090-9957-89e3c1f3e622) (*errors.errorString)
       1: cmd/erasure-sets.go:451:cmd.newErasureSets.func1.1()

API: SYSTEM()
Time: 07:34:57 UTC 04/24/2022
Error: Detected unexpected disk ordering refusing to use the disk - poolID: 1st, found disk mounted at (set=1st, disk=6th) expected mount at (set=1st, disk=3rd): https://minio-tenant-1-pool-0-2.minio-tenant-1-hl.minio-tenant-1.svc.cluster.local:9000/export0(ed1401e4-6ed9-48de-897f-baa0935a003a) (*errors.errorString)
       1: cmd/erasure-sets.go:451:cmd.newErasureSets.func1.1()
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14

找了好久没有找到答案,后来想了下,这个是实验的环境,pv和pvc反复的绑定过很多次了,minio tenant是个有状态的集群,怀疑有关系,在每个节点的目录中,也看到minio相关的系统数据

[root@nccztsjb-node-23 data]# cd minio/
[root@nccztsjb-node-23 minio]# ls
pv1  pv2  pv3  pv4
[root@nccztsjb-node-23 minio]# cd pv1/
[root@nccztsjb-node-23 pv1]# ls
[root@nccztsjb-node-23 pv1]# ls -al
total 0
drwxrwsrwx 3 root      1000  24 Apr 24 10:52 .
drwxrwxrwx 6 root      root  50 Apr 22 17:00 ..
drwxrwsrwx 9 adminuser 1000 124 Apr 25 10:36 .minio.sys
[root@nccztsjb-node-23 pv1]# cat .minio.sys/
cat: .minio.sys/: Is a directory
[root@nccztsjb-node-23 pv1]# ls
[root@nccztsjb-node-23 pv1]# cd .minio.sys/
[root@nccztsjb-node-23 .minio.sys]# ls
buckets  config  format.json  ilm  multipart  pool.bin  tmp  tmp-old
[root@nccztsjb-node-23 .minio.sys]# cd ..
[root@nccztsjb-node-23 pv1]# ls
[root@nccztsjb-node-23 pv1]# pwd
/data/minio/pv1
[root@nccztsjb-node-23 pv1]# 
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21

在每个节点将/data/minio目录删除,重新创建。

解决了该问题。

所以,建议,实验环境,每次的部署都要进行环境的清理!避免出现一些其他的异常情况。


本文转自:https://www.cnblogs.com/chuanzhang053/p/16190774.html

(END)

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/盐析白兔/article/detail/523855
推荐阅读
相关标签
  

闽ICP备14008679号