本文记录使用Velero Restic快速完成云原生应用及PV数据从自建Kubernetes迁移到ACK的实践过程。 此过程也同样适用于其他云厂商Kubernetes集群内的应用及PV数据迁移至ACK。

背景信息

在本示例中， 我们将对自建Kubernetes集群中的一个wordpress应用整体迁移上云到ACK，wordpress示例应用分wordpress和mysql两个组件，
分别绑定两个不同的nfs volume用于应用数据的持久化存储，最后通过NodePort暴露服务。

操作步骤

  1. 步骤一：容器镜像迁移上云
  2. 步骤二：ACK中部署Minio对象存储服务
  3. 步骤三：自建Kubernetes集群中部署示例应用和Velero
  4. 步骤四：在ACK上部署Velero并恢复应用wordpress
  5. 步骤五：验证wordpress应用服务

步骤一：容器镜像迁移上云

自建Kubernetes集群通常位于用户自己的IDC中，容器镜像的存储也会使用自建镜像仓库， 在Kubernetes应用迁移上云之前，您需要先将容器镜像迁移上云到ACR。具体步骤请参见容器镜像迁移。

本示例中，假设wordpress应用中涉及如下容器镜像。 

registry.api.paas.com:5000/admin/wordpress:latest
registry.api.paas.com:5000/admin/mysql:8

迁移上云后的镜像如下。 

registry.cn-hangzhou.aliyuncs.com/ack-migration/wordpress:latest
registry.cn-hangzhou.aliyuncs.com/ack-migration/mysql:8

步骤二：ACK中部署Minio对象存储服务

Minio对象存储服务用于wordpress应用持久化数据卷中数据的迁移中转服务。

  1. 创建命令空间。 
    kubectl create ns minio

  2. 设置您的Minio对象存储服务的用户名密码并替换MINIO_ACCESS_KEY_VALUE MINIO_SECRET_KEY_VALUE的值，部署Minio服务。 
    export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=admin12345
export BUCKET_NAME=velero
sed -i "s/MINIO_ACCESS_KEY_VALUE/$MINIO_ACCESS_KEY/g" minio.yaml
sed -i "s/MINIO_SECRET_KEY_VALUE/$MINIO_SECRET_KEY/g" minio.yaml
sed -i "s/BUCKET/$BUCKET_NAME/g" minio.yaml
kubectl apply -f minio.yaml

    在ACK集群中部署Minio服务并挂载20G的SSD云盘，用于存储中转数据，Minio服务使用LoadBalancer的方式暴露；Minio中新建一个行为Velero的Bucket。

    minio.yaml 内容如下： 

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: storage-pvc
  namespace: minio-test
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: alicloud-disk-ssd
  resources:
    requests:
      storage: 20Gi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: minio-test
  name: minio
  labels:
    component: minio
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: minio
    spec:
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: storage-pvc
      - name: config
        emptyDir: {}
      containers:
      - name: minio
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        args:
        - server
        - /storage
        - --config-dir=/config
        env:
        - name: MINIO_ACCESS_KEY
          value: MINIO_ACCESS_KEY_VALUE
        - name: MINIO_SECRET_KEY
          value: MINIO_SECRET_KEY_VALUE
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"

---
apiVersion: v1
kind: Service
metadata:
  namespace: minio-test
  name: minio
  labels:
    component: minio
spec:
  # ClusterIP is recommended for production environments.
  # Change to NodePort if needed per documentation,
  # but only if you run Minio in a test/trial environment, for example with Minikube.
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    component: minio

 ---
 apiVersion: batch/v1
kind: Job
metadata:
  namespace: minio-test
  name: minio-setup
  labels:
    component: minio
spec:
  template:
    metadata:
      name: minio-setup
    spec:
      restartPolicy: OnFailure
      volumes:
      - name: config
        emptyDir: {}
      containers:
      - name: mc
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - "mc --config-dir=/config config host add BUCKET http://minio:9000 MINIO_ACCESS_KEY_VALUE MINIO_SECRET_KEY_VALUE  && mc --config-dir=/config mb -p BUCKET/BUCKET"
        volumeMounts:
        - name: config
          mountPath: "/config"

  3. 在阿里云容器服务控制台中，单击左侧导航栏中路由与负载均衡 > 服务，选择集群和命名空间。
    在服务列表页面，可以看到刚刚部署Minio对象存储服务。
  4. 单击外部端点，你可以通过IP:Port的方式访问Minio服务。
    Kubernetes应用迁移教程

步骤三：自建Kubernetes集群中部署示例应用和Velero

  1. 自建Kubernetes集群部署wordpress应用。
    说明 本示例中，先创建一个wordpress应用，然后将此应用迁移到ACK，如果您已有需要迁移的应用，跳过此步骤。

    wordpress示例应用分wordpress和mysql两个组件， 分别绑定两个不同的nfs volume用于应用数据的持久化存储，最后通过NodePort暴露服务。yaml文件示例如下：

    # 1. 创建nfs storageclass
$ cat nfs-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs
provisioner: helm.default/nfs
reclaimPolicy: Delete
$ kubectl apply -f  nfs-sc.yaml

# 2. 创建mysql password的secret, echo -n "mysql" |base64
$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysql
type: Opaque
data:
  password: bXlzcWw=
$ kubectl apply -f secret.yaml

# 3. 创建mysql的pvc deployment service
$ cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  type: ClusterIP
  ports:
    - port: 3306
  selector:
    app: mysql
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-volumeclaim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      securityContext:
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999
      containers:
        - image: registry.api.paas.com:5000/admin/mysql:8
          name: mysql
          args:
            - "--default-authentication-plugin=mysql_native_password"
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-volumeclaim
 $ kubectl apply -f mysql.yaml

 # 4. 创建wordpress的pvc deployment service
 $ cat wordpress.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: wordpress
  name: wordpress
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 31570
  selector:
    app: wordpress
  type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wordpress-volumeclaim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
        - image: registry.api.paas.com:5000/admin/wordpress
          name: wordpress
          env:
          - name: WORDPRESS_DB_HOST
            value: mysql:3306
          - name: WORDPRESS_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql
                key: password
          ports:
            - containerPort: 80
              name: wordpress
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
      volumes:
        - name: wordpress-persistent-storage
          persistentVolumeClaim:
            claimName: wordpress-volumeclaim
 $ kubectl apply -f wordpress.yaml

    测试环境绑定hosts后，通过http://wordpress.myk8s.paas.com:31570/访问wordpress应用，确保wordpress应用访问正常。Kubernetes应用迁移教程

  2. 自建Kubernetes集群部署Velero。
    1. 下载Velero客户端工具，并执行如下命令安装和验证Velero客户端。
      本例中下载安装包为 velero-v1.2.0-linux-amd64.tar.gz。 

      $ tar xvf velero-v1.2.0-linux-amd64.tar.gz
$ mv velero-v1.2.0-linux-amd64/velero /usr/bin/velero
$ velero version
Client:
  Version: v1.2.0
  Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
Server:
  Version:

    2. 编辑credentials-velero文件， 设置Minio服务的访问权限信息。 
      [default]
aws_access_key_id=<YourAccessKeyId>
aws_secret_access_key=<YourAccessKeySecret>
      说明 此处的YourAccessKeyId和YourAccessKeySecret为访问Minio服务的登录用户名和密码。

    3. 替换MINIO_SERVER_ADDRESS的值，并使用以下命令部署Velero服务。 
      velero install --provider aws --image registry.cn-hangzhou.aliyuncs.com/acs/velero:v1.2.0 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://MINIO_SERVER_ADDRESS:9000 --use-restic --plugins registry.cn-hangzhou.aliyuncs.com/acs/velero-plugin-for-aws:v1.0.0 --wait

    4. 查看pod的运行状态。 
      kubectl -n velero get po
NAME                      READY   STATUS    RESTARTS   AGE
restic-fqwsc              1/1     Running   0          41s
restic-kfzqt              1/1     Running   0          41s
restic-klxhc              1/1     Running   0          41s
restic-ql2kr              1/1     Running   0          41s
restic-qrsrn              1/1     Running   0          41s
restic-srjmm              1/1     Running   0          41s
velero-67b975f5cb-68nj4   1/1     Running   0          41s

  3. 备份wordpress应用。
    • 如果只需要备份wordpress应用而不备份PV数据，执行以下操作。 
      $ velero backup create wordpress-backup-without-pv --include-namespaces wordpress
Backup request "wordpress-backup-without-pv" submitted successfully.
Run `velero backup describe wordpress-backup-without-pv` or `velero backup logs wordpress-backup-without-pv` for more details.

$ velero backup get
NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
    • 如果需要备份带PV数据的wordpress应用，执行以下操作。 
      # 首先需要为挂载pv数据卷的pod添加annotation， 例如wordpress应用运行了2个pod， 分别为wordpress-7cf5849f47-mbvx4 mysql-74dddbdcc8-h2tls， wordpress-7cf5849f47-mbvx4
# 挂载的volume名为mysql-persistent-storage， mysql-74dddbdcc8-h2tls挂载的volume名为wordpress-persistent-storage, 则添加annotation的命令为
$ kubectl -n wordpress annotate pod/wordpress-7cf5849f47-mbvx4 backup.velero.io/backup-volumes=wordpress-persistent-storage
pod/wordpress-7cf5849f47-mbvx4 annotated
$ kubectl -n wordpress annotate pod/mysql-74dddbdcc8-h2tls backup.velero.io/backup-volumes=mysql-persistent-storage
pod/mysql-74dddbdcc8-h2tls annotated

# 备份wordpress
$ velero backup create wordpress-backup-with-pv --include-namespaces wordpress
Backup request "wordpress-backup-with-pv" submitted successfully.
Run `velero backup describe wordpress-backup-with-pv` or `velero backup logs wordpress-backup-with-pv` for more details.
$ velero backup get
NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-with-pv      Completed   2019-12-12 14:23:40 +0800 CST   29d       default            <none>
wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
    查看Minio服务器可以看到备份的文件。Kubernetes应用迁移教程

步骤四：在ACK上部署Velero并恢复应用wordpress

  1. 在ACK部署Velero。
    操作步骤同自建Kubernetes集群部署Velero。

    部署完成后，查看backup列表的运行状态。 

    $ velero backup get
NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-with-pv      Completed   2019-12-12 14:23:40 +0800 CST   29d       default            <none>
wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>

  2. 在ACK中恢复wordpress应用。
    wordpress应用使用NFS类型持久化数据卷，相应的，在ACK中我们可以适配NAS volume，在本示例中，我们创建与Wordpress应用所使用的StorageClass
    NFS， 但后端存储介质使用SSD云盘块存储。

    本示例使用ACK集群使用CSI plugin， 请参见动态云盘卷。

    1. 创建StorageClass。
      如果在ACK上备份wordpress应用不带pv数据，则跳过此步骤。 

      $ cat nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: nfs
provisioner: diskplugin.csi.alibabacloud.com
parameters:
    type: cloud_ssd
reclaimPolicy: Retain

$ kubectl apply -f nfs.yaml
storageclass.storage.k8s.io/nfs created

    2. 恢复wordpress应用。 
      $ velero restore create --from-backup wordpress-backup-with-pv
$ velero restore get
NAME                                      BACKUP                     STATUS       WARNINGS   ERRORS   CREATED                         SELECTOR
wordpress-backup-with-pv-20191212152745   wordpress-backup-with-pv   InProgress   0          0        2019-12-12 15:27:45 +0800 CST   <none>
$ velero restore get
      此时查看wordpress应用运行情况，会有镜像拉取失败的问题。 

      $ kubectl -n wordpress get po
NAME                         READY   STATUS         RESTARTS   AGE
mysql-669b4666cd-trsnz       0/1     ErrImagePull   0          19m
mysql-74dddbdcc8-h2tls       0/1     Init:0/1       0          19m
wordpress-7cf5849f47-mbvx4   0/1     Init:0/1       0          19m
wordpress-bb5d74d95-xcjxw    0/1     ErrImagePull   0          19m

    3. 编辑Deployment把image字段替换成步骤一：容器镜像迁移上云中迁移后的镜像地址。 
      # edit 编辑deployment并修改image url
$ kubectl -n wordpress edit deployment mysql
$ kubectl -n wordpress edit deployment wordpress

步骤五：验证wordpress应用服务

查看wordpress应用运行情况。 

$ kubectl -n wordpress get po
NAME                         READY   STATUS    RESTARTS   AGE
mysql-678b5d8499-vckfd       1/1     Running   0          100s
wordpress-8566f5f7d8-7shk6   1/1     Running   0          3m18s

测试环境绑定hosts后，通过http://wordpress.myk8s.paas.com:31570/访问wordpress应用。Kubernetes应用迁移教程