kubernetes集群备份与恢复管理工具Velero_velero备份

一、Velero简介

Velero 实现的 kubernetes 资源备份能力，可以轻松实现 Kubernetes 集群的数据备份和恢复、复制 kubernetes 集群资源到其他 kubernetes 集群或者快速复制生产环境到测试环境等功能，这种备份就类似于把资源的 yaml 文件进行整体备份，从而保障资源的完整性。

Velero 对存储的支持较好，可以支持很多种存储资源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等.

二、Velero工作流程

Velero 客户端调用Kubernetes API Server创建Backup任务。Backup 控制器基于watch 机制通过API Server获取到备份任务。Backup 控制器开始执行备份动作，其会通过请求API Server获取需要备份的数据。Backup 控制器将获取到的数据备份到指定的对象存储server端。

三、Velero备份过程

• 本地 Velero 客户端发送备份指令。

• Kubernetes 集群内就会创建一个 Backup 对象。

• BackupController 监测 Backup 对象并开始备份过程。

• BackupController 会向 API Server 查询相关数据。

• BackupController 将查询到的数据备份到远端的对象存储。

四、Velero特性

Velero 目前包含以下特性：

• 支持 Kubernetes 集群数据备份和恢复

• 支持复制当前 Kubernetes 集群的资源到其它 Kubernetes 集群

• 支持复制生产环境到开发以及测试环境

五、Velero组件

Velero 组件一共分两部分，分别是服务端和客户端。

• 服务端：运行在你 Kubernetes 的集群中

• 客户端：是一些运行在本地的命令行的工具，需要已配置好 kubectl 及集群 kubeconfig 的机器上

六、Velero支持备份存储

• AWS S3 以及兼容 S3 的存储，比如：Minio

• Azure BloB 存储

• Google Cloud 存储

• 阿里云OSS

七、Velero适用场景

• 灾备场景：提供备份恢复k8s集群的能力

• 迁移场景：提供拷贝集群资源到其他集群的能力（复制同步开发，测试，生产环境的集群配置，简化环境配置）

八、Velero备份与etcd备份的区别

• 与 Etcd 备份相比，直接备份 Etcd 是将集群的全部资源备份起来。

• Velero 可以对 Kubernetes 集群内对象级别进行备份。

• 除了对 Kubernetes 集群进行整体备份外，Velero 还可以通过对 Type、Namespace、Label 等对象进行分类备份或者恢复。

注意：备份过程中创建的对象是不会被备份的。

九、Velero部署 

9.1 Velero工具获取

github上有

[root@k8s-master01 ~]# wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.0/velero-v1.11.0-linux-amd64.tar.gz

9.2 部署Velero依赖后端存储MinIo

[root@k8s-master01 ~]# tar xf velero-v1.11.0-linux-amd64.tar.gz

[root@k8s-master01 ~]# ls
velero-v1.11.0-linux-amd64 

[root@k8s-master01 ~]# cd velero-v1.11.0-linux-amd64/

[root@k8s-master01 velero-v1.11.0-linux-amd64]# ls
examples  LICENSE  velero

[root@k8s-master01 velero-v1.11.0-linux-amd64]# cd examples/

[root@k8s-master01 examples]# ls
minio  nginx-app

[root@k8s-master01 examples]# cd minio/
[root@k8s-master01 minio]# ls
00-minio-deployment.yaml 

[root@k8s-master01 minio]# cat 00-minio-deployment.yaml
# Copyright 2017 the Velero contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
apiVersion: v1
kind: Namespace
metadata:
  name: velero

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: velero
  name: minio
  labels:
    component: minio
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: minio
  template:
    metadata:
      labels:
        component: minio
    spec:
      volumes:
      - name: storage
        emptyDir: {}
      - name: config
        emptyDir: {}
      containers:
      - name: minio
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        args:
        - server
        - /storage
        - --config-dir=/config
        - --console-address
        - ":30605"
        env:
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        ports:
        - name: web
          containerPort: 9000
        - name: console
          containerPort: 30605
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"

---
apiVersion: v1
kind: Service
metadata:
  namespace: velero
  name: minio
  labels:
    component: minio
spec:
  # ClusterIP is recommended for production environments.
  # Change to NodePort if needed per documentation,
  # but only if you run Minio in a test/trial environment, for example with Minikube.
  type: NodePort
  ports:
    - name: web
      port: 9000
      targetPort: 9000
      nodePort: 31900
      protocol: TCP
    - name: console
      port: 30605
      targetPort: 30605
      nodePort: 30605
      protocol: TCP
  selector:
    component: minio

---
apiVersion: batch/v1
kind: Job
metadata:
  namespace: velero
  name: minio-setup
  labels:
    component: minio
spec:
  template:
    metadata:
      name: minio-setup
    spec:
      restartPolicy: OnFailure
      volumes:
      - name: config
        emptyDir: {}
      containers:
      - name: mc
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
        volumeMounts:
        - name: config
          mountPath: "/config"

 

[root@k8s-master01 minio]# pwd
/root/velero-v1.11.0-linux-amd64/examples/minio

[root@k8s-master01 minio]# ls
00-minio-deployment.yaml

[root@k8s-master01 minio]# kubectl apply -f 00-minio-deployment.yaml
namespace/velero created
deployment.apps/minio created
service/minio created
job.batch/minio-setup created

 

注意：访问用户名为minio，密码为minio123

[root@k8s-master01 minio]# kubectl get all -n velero
NAME                         READY   STATUS      RESTARTS   AGE
pod/minio-6959795f76-fc4ls   1/1     Running     0          2m52s
pod/minio-setup-fsfvm        0/1     Completed   0          2m51s

NAME            TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)                          AGE
service/minio   NodePort   10.96.2.7    <none>        9000:31900/TCP,30605:30605/TCP   2m52s
 

 

 

9.3 安装Velero 

9.3.1 创建访问minio密钥

[root@k8s-master01 velero-v1.11.0-linux-amd64]# vim cred-velero
[root@k8s-master01 velero-v1.11.0-linux-amd64]# cat cred-velero
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

9.3.2 安装Velero客户端到当前系统 

[root@k8s-master01 velero-v1.11.0-linux-amd64]# pwd
/root/velero-v1.11.0-linux-amd64
[root@k8s-master01 velero-v1.11.0-linux-amd64]# ls
examples  LICENSE  velero
[root@k8s-master01 velero-v1.11.0-linux-amd64]# cp velero /usr/bin/
[root@k8s-master01 velero-v1.11.0-linux-amd64]# ls /usr/bin/velero
/usr/bin/velero

9.3.3 在K8S集群快速安装Velero 

[root@k8s-master01 ~]# velero completion bash

 

[root@k8s-master01 velero-v1.11.0-linux-amd64]# ls
cred-velero  examples  LICENSE  velero  velero.sh
[root@k8s-master01 velero-v1.11.0-linux-amd64]# vim velero.sh
[root@k8s-master01 velero-v1.11.0-linux-amd64]# cat velero.sh
velero install \
     --provider aws \
     --plugins velero/velero-plugin-for-aws:v1.0.0 \
     --bucket velero \
     --secret-file ./cred-velero \
     --use-volume-snapshots=false \
     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc.cluster.local.:9000 

[root@k8s-master01 velero-v1.11.0-linux-amd64]# sh velero.sh 

[root@k8s-master01 velero-v1.11.0-linux-amd64]# kubectl get pods -n velero
NAME                      READY   STATUS      RESTARTS   AGE
minio-6959795f76-fc4ls    1/1     Running     0          38m
minio-setup-fsfvm         0/1     Completed   0          38m
velero-84fff5d559-rr6bw   1/1     Running     0          93s 

十、velero命令介绍 

[root@k8s-master01 ~]# velero create backup --help
Create a backup

Usage:
  velero create backup NAME [flags]

Examples:
  # Create a backup containing all resources.
  velero backup create backup1

  # Create a backup including only the nginx namespace.
  velero backup create nginx-backup --include-namespaces nginx

  # Create a backup excluding the velero and default namespaces.
  velero backup create backup2 --exclude-namespaces velero,default

  # Create a backup based on a schedule named daily-backup.
  velero backup create --from-schedule daily-backup

  # View the YAML for a backup that doesn't snapshot volumes, without sending it to the server.
  velero backup create backup3 --snapshot-volumes=false -o yaml

  # Wait for a backup to complete before returning from the command.
  velero backup create backup4 --wait

# 剔除 namespace
--exclude-namespaces stringArray                  namespaces to exclude from the backup

# 剔除资源类型
--exclude-resources stringArray                   resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io

# 包含集群资源类型 
--include-cluster-resources optionalBool[=true]   include cluster-scoped resources in the backup

# 包含 namespace
--include-namespaces stringArray                  namespaces to include in the backup (use '*' for all namespaces) (default *)

# 包含 namespace 资源类型
--include-resources stringArray                   resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)

# 给这个备份加上标签
--labels mapStringString                          labels to apply to the backup
-o, --output string                               Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.

# 对指定标签的资源进行备份
-l, --selector labelSelector                      only back up resources matching this label selector (default <none>)

# 对 PV 创建快照
--snapshot-volumes optionalBool[=true]            take snapshots of PersistentVolumes as part of the backup

# 指定备份的位置
--storage-location string                         location in which to store the backup

# 备份数据多久删掉

--ttl duration                                    how long before the backup can be garbage collected (default 720h0m0s)

# 指定快照的位置，也就是哪一个公有云驱动
--volume-snapshot-locations strings               list of locations (at most one per provider) where volume snapshots should be stored

 

十一、使用velero实现k8s资源对象备份 

11.1 创建用于备份测试应用

[root@k8s-master01 ~]# kubectl apply -f velero-v1.11.0-linux-amd64/examples/nginx-app/base.yaml
namespace/nginx-example created
deployment.apps/nginx-deployment created
service/my-nginx created 

 

11.2 创建备份 

[root@k8s-master01 ~]# velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.

 

11.3 查看备份 

[root@k8s-master01 ~]# velero backup describe nginx-backup
Name:         nginx-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.25.5
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25

Phase:  Completed

Namespaces:
  Included:  nginx-example
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-06-08 10:49:07 +0800 CST
Completed:  2023-06-08 10:49:08 +0800 CST

Expiration:  2023-07-08 10:49:07 +0800 CST

Velero-Native Snapshots: <none included>

查看备份位置
[root@k8s-master01 ~]# velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        velero          Available   2023-06-09 10:16:49 +0800 CST   ReadWrite     true 

查看备份文件
[root@k8s-master01 ~]# kubectl get backups.velero.io -n velero
NAME                 AGE
nginx-backup         23h 

 

十二、使用velero实现对K8S资源对象进行恢复 

12.1 删除备份测试应用 

[root@k8s-master01 ~]# kubectl delete -f velero-v1.11.0-linux-amd64/examples/nginx-app/base.yaml
namespace "nginx-example" deleted
deployment.apps "nginx-deployment" deleted
service "my-nginx" deleted

 

12.2 恢复备份测试应用 

[root@k8s-master01 ~]# velero restore create --from-backup nginx-backup --wait
Restore request "nginx-backup-20230608111349" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.

Restore completed with status: Completed. You may check for more information using the commands `velero restore describe nginx-backup-20230608111349` and `velero restore logs nginx-backup-20230608111349`.

十三、周期性备份任务 

# Create a backup every 6 hours
velero create schedule NAME --schedule="0 */6 * * *"

# Create a backup every 6 hours with the @every notation
velero create schedule NAME --schedule="@every 6h"

# Create a daily backup of the web namespace
velero create schedule NAME --schedule="@every 24h" --include-namespaces web

# Create a weekly backup, each living for 90 days (2160 hours)
velero create schedule NAME --schedule="@every 168h" --ttl 2160h0m0s

# 每日对anchnet-devops-dev/anchnet-devops-test/anchnet-devops-prod/xxxxx-devops-common-test 名称空间进行备份
velero create schedule anchnet-devops-dev --schedule="@every 24h" --include-namespaces xxxxx-devops-dev 
velero create schedule anchnet-devops-test --schedule="@every 24h" --include-namespaces xxxxx-devops-test
velero create schedule anchnet-devops-prod --schedule="@every 24h" --include-namespaces xxxxx-devops-prod 
velero create schedule anchnet-devops-common-test --schedule="@every 24h" --include-namespaces xxxxx-devops-common-test

案例：
[root@k8s-master01 ~]# velero create schedule nginx-backups --schedule="0 */1 * * *" --include-namespaces nginx-example
Schedule "nginx-backups" created successfully.

[root@k8s-master01 ~]# velero get schedules
NAME            STATUS    CREATED                         SCHEDULE      BACKUP TTL   LAST BACKUP   SELECTOR   PAUSED
nginx-backups   Enabled   2023-06-09 10:36:08 +0800 CST   0 */1 * * *   0s           n/a           <none>     false

 

注意事项：

• 在velero备份的时候，备份过程中创建的对象是不会被备份的。

• velero restore 恢复不会覆盖已有的资源，只恢复当前集群中不存在的资源。已有的资源不会回滚到之前的版本，如需要回滚，需在restore之前提前删除现有的资源。

• velero也可作为一个crontjob来运行，定期备份数据。

十四、在其它K8S集群中部署velero及恢复应用

14.1 在其它K8S集群中部署velero

从第一个K8S集群中复制文件到第二个K8S集群master节点
[root@k8s-master01 ~]# scp -r velero-v1.11.0-linux-amd64 192.168.10.160:/root

 

[root@k8s-master01 ~]# cp velero-v1.11.0-linux-amd64/velero /usr/bin/ 

[root@k8s-master01 velero-v1.11.0-linux-amd64]# cat cred-velero
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

[root@k8s-master01 velero-v1.11.0-linux-amd64]# cat velero.sh
velero install \
     --provider aws \
     --plugins velero/velero-plugin-for-aws:v1.0.0 \
     --bucket velero \
     --secret-file ./cred-velero \
     --use-volume-snapshots=false \
     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc.cluster.local.:9000
     
     
[root@k8s-master01 velero-v1.11.0-linux-amd64]# vim velero.sh
[root@k8s-master01 velero-v1.11.0-linux-amd64]# cat velero.sh
velero install \
     --provider aws \
     --plugins velero/velero-plugin-for-aws:v1.0.0 \
     --bucket velero \
     --secret-file ./cred-velero \
     --use-volume-snapshots=false \
     --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.10.140:31900

 

[root@k8s-master01 velero-v1.11.0-linux-amd64]# sh velero.sh 

14.2 恢复应用到新K8S集群 

查看备份位置 
[root@k8s-master01 velero-v1.11.0-linux-amd64]# velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE       LAST VALIDATED                  ACCESS MODE   DEFAULT
default   aws        velero          Available   2023-06-09 10:16:44 +0800 CST   ReadWrite     true

查看备份文件
[root@k8s-master01 velero-v1.11.0-linux-amd64]# kubectl get backups.velero.io -n velero
NAME                 AGE
nginx-backup         4m7s 

恢复备份到新K8S集群
[root@k8s-master01 ~]# velero restore create \
--namespace velero \
--from-backup nginx-backup --wait 

查看备份恢复情况

[root@k8s-master01 ~]# kubectl get ns
NAME               STATUS   AGE
calico-apiserver   Active   56d
calico-system      Active   56d
default            Active   56d
kube-node-lease    Active   56d
kube-public        Active   56d
kube-system        Active   56d
nginx-example      Active   7s
tigera-operator    Active   56d
velero             Active   6m17s

[root@k8s-master01 ~]# kubectl get pods -n nginx-example
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-747864f4b5-5r7kd   1/1     Running   0          22s
nginx-deployment-747864f4b5-9n5gz   1/1     Running   0          22s