kubernetes的备份方案

采用本地minikube进行验证，安装方式如下

curl -Lo minikube http://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 && chmod +x minikube 
sudo apt-get install conntrack
# 使用了本机的docker
minikube start --vm-driver=none --registry-mirror=https://registry.docker-cn.com

集群资源备份

velero

Velero 是一个云原生的灾难恢复和迁移工具，它本身也是开源的, 采用 Go 语言编写，可以安全的备份、恢复和迁移Kubernetes集群资源和持久卷

Velero 支持两种关于后端存储的 CRD，分别是 BackupStorageLocation 和 VolumeSnapshotLocation
BackupStorageLocation 主要用来定义 Kubernetes 集群资源的数据存放位置，也就是集群对象数据，不是 PVC 的数据。主要支持的后端存储是 S3 兼容的存储，比如：Mino 和阿里云 OSS 等
VolumeSnapshotLocation 主要用来给 PV 做快照，需要云提供商提供插件。阿里云已经提供了插件，这个需要使用 CSI 等存储机制。你也可以使用专门的备份工具 Restic，把 PV 数据备份到阿里云 OSS 中去(安装时需要自定义选项)

备份过程

本地 Velero 客户端发送备份指令
Kubernetes 集群内就会创建一个 Backup 对象
BackupController 监测 Backup 对象并开始备份过程
BackupController 会向 API Server 查询相关数据
BackupController 将查询到的数据备份到远端的对象存储

安装

下载velero
下载最新版本
https://github.com/vmware-tanzu/velero/releases 解压获取二进制程序

安装minio
在解压velero目录的example/minio下
kubectl create -f examples/minio/00-minio-deployment.yaml
svc开放nodeport访问

---
apiVersion: v1
kind: Namespace
metadata:
  name: velero

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: velero
  name: minio
  labels:
    component: minio
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: minio
  template:
    metadata:
      labels:
        component: minio
    spec:
      volumes:
      - name: storage
        emptyDir: {}
      - name: config
        emptyDir: {}
      containers:
      - name: minio
        image: minio/minio:RELEASE.2021-01-05T05-22-38Z
        imagePullPolicy: IfNotPresent
        args:
        - server
        - /storage
        - --config-dir=/config
        env:
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"

---
apiVersion: v1
kind: Service
metadata:
  namespace: velero
  name: minio
  labels:
    component: minio
spec:
  # ClusterIP is recommended for production environments.
  # Change to NodePort if needed per documentation,
  # but only if you run Minio in a test/trial environment, for example with Minikube.
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    component: minio

---
apiVersion: batch/v1
kind: Job
metadata:
  namespace: velero
  name: minio-setup
  labels:
    component: minio
spec:
  template:
    metadata:
      name: minio-setup
    spec:
      restartPolicy: OnFailure
      volumes:
      - name: config
        emptyDir: {}
      containers:
      - name: mc
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
        volumeMounts:
        - name: config
          mountPath: "/config"

创建mino凭证
vi examples/minio/credentials-velero

1
2
3

[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

集群创建velero资源

# s3Url指的是mini访问地址
velero install \
   --provider aws \
   --bucket velero \
   --secret-file /home/backup/minio-credentials-velero \
   --use-volume-snapshots=false \
   --plugins velero/velero-plugin-for-aws:v1.1.0 \
   --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.221.2.186:30532

如果配置错误需要先删除再重新install，避免受到上次安装配置文件的影响

备份集群资源

备份命令解析

$ velero create backup NAME [flags]

# 剔除 namespace
--exclude-namespaces stringArray                  namespaces to exclude from the backup

# 剔除资源类型
--exclude-resources stringArray                   resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io

# 包含集群资源类型 
--include-cluster-resources optionalBool[=true]   include cluster-scoped resources in the backup

# 包含 namespace
--include-namespaces stringArray                  namespaces to include in the backup (use '*' for all namespaces) (default *)

# 包含 namespace 资源类型
--include-resources stringArray                   resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)

# 给这个备份加上标签
--labels mapStringString                          labels to apply to the backup
-o, --output string                               Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.

# 对指定标签的资源进行备份
-l, --selector labelSelector                      only back up resources matching this label selector (default <none>)

# 对 PV 创建快照
--snapshot-volumes optionalBool[=true]            take snapshots of PersistentVolumes as part of the backup

# 指定备份的位置
--storage-location string                         location in which to store the backup

# 备份数据多久删掉

--ttl duration                                    how long before the backup can be garbage collected (default 720h0m0s)

# 指定快照的位置，也就是哪一个公有云驱动
--volume-snapshot-locations strings               list of locations (at most one per provider) where volume snapshots should be stored

备份操作
测试应用可以拿example目录下的nginx/base.yaml进行创建

# 备份k8s namespace下的资源
velero backup create test-backup1 --snapshot-volumes=false  --include-namespaces nginx-example1

# 查看备份
velero backup get
velero backup describe test-backup1                                    
velero backup download test-backup1    

# 定时备份(每天1点 保留7天)
velero create schedule test-projec-schedule --schedule="0 1 * * *" --ttl 168h --include-namespaces test-project

# 恢复
velero restore create --from-backup test-backup1

登陆minio可以发现相关文件都存储在velero bucket下
注意: 1 备份过程中创建的对象是不会被备份的; 2 已经存在且更新的资源不会被还原成备份前的状态，如果要恢复则需要删除再还原

持久卷备份

restic

Restic 是一款 GO 语言开发的数据加密备份工具，顾名思义，可以将本地数据加密后传输到指定的仓库。支持的仓库有 Local、SFTP、Aws S3、Minio、OpenStack Swift、Backblaze B2、Azure BS、Google Cloud storage、Rest Server

安装

下载程序
https://github.com/restic/restic/releases 下载二进制程序
使用 bzip2 -d name.bz2 解压
初始化repo(使用minio 做存储)
$ export AWS_ACCESS_KEY_ID=minio
$ export AWS_SECRET_ACCESS_KEY=minio123

./restic -r s3:http://minio.dash/restic init
restic -r s3:http://10.221.2.186:30532/restic init

初始化期间需要输入密码，这个密码后续执行命令都需要用到

进行备份

# 备份目录
./restic -r s3:http://minio.dash/restic --verbose backup /tmp/hostpath-provisioner/nginx-example/nginx-logs/

# 查看备份
./restic -r s3:http://minio.dash/restic snapshots
./restic -r s3:http://minio.dash/restic ls 20c43b47

# 备份恢复
./restic -r s3:http://minio.dash/restic restore 363e2805 -t /tmp/backup/

velero 整合restic 进行存储卷和集群资源备份(未测试成功)

velero的 volumeSnapshotLocation
主要用来给 PV 做快照，需要云提供商提供插件。各大厂商已经提供相关了插件，这个需要使用 CSI 等存储机制
支持的插件列表https://velero.io/plugins/

备份pv数据需要云厂商支持，参考：
https://blog.csdn.net/easylife206/article/details/102927512
https://blog.51cto.com/kaliarch/2531077?source=drh

velero 需要开启存储快照功能

velero install \
   --provider aws \
   --bucket velero \
   --secret-file /home/backup/minio-credentials-velero \
   --use-volume-snapshots=true \
   --plugins velero/velero-plugin-for-aws:v1.1.0 \
   --use-restic \
   --snapshot-location-config region=minio \
   --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.221.2.186:30532

测试应用可以拿example目录下的nginx/with-pvc.yaml进行创建
使用 Restic 给带有 PVC 的 Pod 进行备份，必须先给 Pod 加上注解
kubectl -n test-velero annotate pod nfs-pvc-7d75fbbcdf-dn7xw backup.velero.io/backup-volumes=www

备份带存储卷的应用资源
velero backup create pvc-backup –snapshot-volumes –include-namespaces nginx-example

进行恢复
velero restore create –from-backup ppvc-backup –restore-volumes

备份原理：https://velero.io/docs/v1.5/restic/#troubleshooting

参考

https://www.imooc.com/article/310069

https://www.cnblogs.com/zphqq/p/13155394.html