【Milvus】记录一次基于milvus-backup做的Milvus备份与恢复

环境

milvus：v2.2.4

go：1.20.2 darwin/amd64

milvus-backup：v0.2.2

代码准备

https://github.com/zilliztech/milvus-backup/releases

如果你的milvus是2.2.9版本及以上，可以直接下载最新的版本：git clone https://github.com/zilliztech/milvus-backup.git

默认使用的配置文件在config目录下，如果不需要手动指定，直接修改改文件即可，主要修改milvus和minio相关配置即可

示例：

# Configures the system log output.
log:
  level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
  console: true
  file:
    rootPath: "logs/backup.log"

http:
  simpleResponse: true

# milvus proxy address, compatible to milvus.yaml
milvus:
  address: milvus-t-milvus.milvus.svc.t2.test.xdf.cn
  port: 19530
  authorizationEnabled: false
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  tlsMode: 0
  user: xxxx
  password: xxxx

# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  address: minio.openai.svc.t2.test.xdf.cn # Address of MinIO/S3
  port: 80   # Port of MinIO/S3
  accessKeyID: xxxx # accessKeyID of MinIO/S3
  secretAccessKey: xxxx # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  cloudProvider: "aws"
  iamEndpoint: ""

  bucketName: "milvus-t" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "files" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  backupBucketName: "milvus-t-backup-all" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "files" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

backup:
  maxSegmentGroupSize: 2G
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

备份

构建/运行

进入项目根目录，依次执行下面的命令：

go get
go build
1
2

会在根目录下生成可执行文件：

Zhurunhua-2:milvus-backup-0.2.2 xdf$ ./milvus-backup help
milvus-backup is a backup tool for milvus.

Usage:
  milvus-backup [flags]
  milvus-backup [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  create      create subcommand create a backup.
  delete      delete subcommand delete backup by name.
  get         get subcommand get backup by name.
  help        Help about any command
  list        list subcommand shows all backup in the cluster.
  restore     restore subcommand restore a backup.
  server      server subcommand start milvus-backup RESTAPI server.
  version     print the version of Milvus backup tool

Flags:
      --config string   config YAML file of milvus (default "backup.yaml")
  -h, --help            help for milvus-backup

Use "milvus-backup [command] --help" for more information about a command.
Zhurunhua-2:milvus-backup-0.2.2 xdf$ 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

备份需要使用create命令：

示例：指定Collection进行备份

./milvus-backup create --colls tk_td_question_v2 -n tiku_backup
1

总共19G数据，大概几分钟就完事了

验证

从minio-console可以看到，对应的备份桶已经创建了，数据也备份好了：

binlogs中和源文件一直，meta中会生成一些元数据：

恢复

TODO

遇到的问题

我刚开始下载的是最新的源码

1. this version of sdk is incompatible with server, please downgrade your sdk or upgrade your server

解决：milvus版本过低，与源码所支持的不一致，下载非最新版本的milvus-backup解决。