本文记录使用Velero Restic快速完成云原生应用及PV数据从自建Kubernetes迁移到ACK的实践过程。 此过程也同样适用于其他云厂商Kubernetes集群内的应用及PV数据迁移至ACK。

背景信息

在本示例中, 我们将对自建Kubernetes集群中的一个wordpress应用整体迁移上云到ACK,wordpress示例应用分wordpress和mysql两个组件,
分别绑定两个不同的nfs volume用于应用数据的持久化存储,最后通过NodePort暴露服务。

操作步骤

  1. 步骤一:容器镜像迁移上云
  2. 步骤二:ACK中部署Minio对象存储服务
  3. 步骤三:自建Kubernetes集群中部署示例应用和Velero
  4. 步骤四:在ACK上部署Velero并恢复应用wordpress
  5. 步骤五:验证wordpress应用服务

步骤一:容器镜像迁移上云

自建Kubernetes集群通常位于用户自己的IDC中,容器镜像的存储也会使用自建镜像仓库, 在Kubernetes应用迁移上云之前,您需要先将容器镜像迁移上云到ACR。具体步骤请参见容器镜像迁移。

本示例中,假设wordpress应用中涉及如下容器镜像。

registry.api.paas.com:5000/admin/wordpress:latest
registry.api.paas.com:5000/admin/mysql:8

迁移上云后的镜像如下。

registry.cn-hangzhou.aliyuncs.com/ack-migration/wordpress:latest
registry.cn-hangzhou.aliyuncs.com/ack-migration/mysql:8

步骤二:ACK中部署Minio对象存储服务

Minio对象存储服务用于wordpress应用持久化数据卷中数据的迁移中转服务。

  1. 创建命令空间。
    kubectl create ns minio

  2. 设置您的Minio对象存储服务的用户名密码并替换MINIO_ACCESS_KEY_VALUE MINIO_SECRET_KEY_VALUE的值,部署Minio服务。
    export MINIO_ACCESS_KEY=admin
    export MINIO_SECRET_KEY=admin12345
    export BUCKET_NAME=velero
    sed -i "s/MINIO_ACCESS_KEY_VALUE/$MINIO_ACCESS_KEY/g" minio.yaml
    sed -i "s/MINIO_SECRET_KEY_VALUE/$MINIO_SECRET_KEY/g" minio.yaml
    sed -i "s/BUCKET/$BUCKET_NAME/g" minio.yaml
    kubectl apply -f minio.yaml

    在ACK集群中部署Minio服务并挂载20G的SSD云盘,用于存储中转数据,Minio服务使用LoadBalancer的方式暴露;Minio中新建一个行为Velero的Bucket。

    minio.yaml 内容如下:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: storage-pvc
      namespace: minio-test
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: alicloud-disk-ssd
      resources:
        requests:
          storage: 20Gi
    
    ---
    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
      namespace: minio-test
      name: minio
      labels:
        component: minio
    spec:
      strategy:
        type: Recreate
      template:
        metadata:
          labels:
            component: minio
        spec:
          volumes:
          - name: storage
            persistentVolumeClaim:
              claimName: storage-pvc
          - name: config
            emptyDir: {}
          containers:
          - name: minio
            image: minio/minio:latest
            imagePullPolicy: IfNotPresent
            args:
            - server
            - /storage
            - --config-dir=/config
            env:
            - name: MINIO_ACCESS_KEY
              value: MINIO_ACCESS_KEY_VALUE
            - name: MINIO_SECRET_KEY
              value: MINIO_SECRET_KEY_VALUE
            ports:
            - containerPort: 9000
            volumeMounts:
            - name: storage
              mountPath: "/storage"
            - name: config
              mountPath: "/config"
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      namespace: minio-test
      name: minio
      labels:
        component: minio
    spec:
      # ClusterIP is recommended for production environments.
      # Change to NodePort if needed per documentation,
      # but only if you run Minio in a test/trial environment, for example with Minikube.
      type: LoadBalancer
      ports:
        - port: 9000
          targetPort: 9000
          protocol: TCP
      selector:
        component: minio
    
     ---
     apiVersion: batch/v1
    kind: Job
    metadata:
      namespace: minio-test
      name: minio-setup
      labels:
        component: minio
    spec:
      template:
        metadata:
          name: minio-setup
        spec:
          restartPolicy: OnFailure
          volumes:
          - name: config
            emptyDir: {}
          containers:
          - name: mc
            image: minio/mc:latest
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - "mc --config-dir=/config config host add BUCKET http://minio:9000 MINIO_ACCESS_KEY_VALUE MINIO_SECRET_KEY_VALUE  && mc --config-dir=/config mb -p BUCKET/BUCKET"
            volumeMounts:
            - name: config
              mountPath: "/config"

  3. 在阿里云容器服务控制台中,单击左侧导航栏中路由与负载均衡 > 服务,选择集群和命名空间。
    在服务列表页面,可以看到刚刚部署Minio对象存储服务。
  4. 单击外部端点,你可以通过IP:Port的方式访问Minio服务。
    Kubernetes应用迁移教程

步骤三:自建Kubernetes集群中部署示例应用和Velero

  1. 自建Kubernetes集群部署wordpress应用。
    说明 本示例中,先创建一个wordpress应用,然后将此应用迁移到ACK,如果您已有需要迁移的应用,跳过此步骤。

    wordpress示例应用分wordpress和mysql两个组件, 分别绑定两个不同的nfs volume用于应用数据的持久化存储,最后通过NodePort暴露服务。yaml文件示例如下:

    # 1. 创建nfs storageclass
    $ cat nfs-sc.yaml
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: nfs
    provisioner: helm.default/nfs
    reclaimPolicy: Delete
    $ kubectl apply -f  nfs-sc.yaml
    
    # 2. 创建mysql password的secret, echo -n "mysql" |base64
    $ cat secret.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: mysql
    type: Opaque
    data:
      password: bXlzcWw=
    $ kubectl apply -f secret.yaml
    
    # 3. 创建mysql的pvc deployment service
    $ cat mysql.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: mysql
      labels:
        app: mysql
    spec:
      type: ClusterIP
      ports:
        - port: 3306
      selector:
        app: mysql
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: mysql-volumeclaim
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs"
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
    
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: mysql
      labels:
        app: mysql
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: mysql
      template:
        metadata:
          labels:
            app: mysql
        spec:
          securityContext:
            runAsUser: 999
            runAsGroup: 999
            fsGroup: 999
          containers:
            - image: registry.api.paas.com:5000/admin/mysql:8
              name: mysql
              args:
                - "--default-authentication-plugin=mysql_native_password"
              env:
                - name: MYSQL_ROOT_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: mysql
                      key: password
              ports:
                - containerPort: 3306
                  name: mysql
              volumeMounts:
                - name: mysql-persistent-storage
                  mountPath: /var/lib/mysql
          volumes:
            - name: mysql-persistent-storage
              persistentVolumeClaim:
                claimName: mysql-volumeclaim
     $ kubectl apply -f mysql.yaml
    
     # 4. 创建wordpress的pvc deployment service
     $ cat wordpress.yaml
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: wordpress
      name: wordpress
    spec:
      ports:
        - port: 80
          targetPort: 80
          protocol: TCP
          nodePort: 31570
      selector:
        app: wordpress
      type: NodePort
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: wordpress-volumeclaim
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfs"
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: wordpress
      labels:
        app: wordpress
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: wordpress
      template:
        metadata:
          labels:
            app: wordpress
        spec:
          containers:
            - image: registry.api.paas.com:5000/admin/wordpress
              name: wordpress
              env:
              - name: WORDPRESS_DB_HOST
                value: mysql:3306
              - name: WORDPRESS_DB_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: mysql
                    key: password
              ports:
                - containerPort: 80
                  name: wordpress
              volumeMounts:
                - name: wordpress-persistent-storage
                  mountPath: /var/www/html
          volumes:
            - name: wordpress-persistent-storage
              persistentVolumeClaim:
                claimName: wordpress-volumeclaim
     $ kubectl apply -f wordpress.yaml

    测试环境绑定hosts后,通过http://wordpress.myk8s.paas.com:31570/访问wordpress应用,确保wordpress应用访问正常。Kubernetes应用迁移教程

  2. 自建Kubernetes集群部署Velero。
    1. 下载Velero客户端工具,并执行如下命令安装和验证Velero客户端。
      本例中下载安装包为 velero-v1.2.0-linux-amd64.tar.gz

      $ tar xvf velero-v1.2.0-linux-amd64.tar.gz
      $ mv velero-v1.2.0-linux-amd64/velero /usr/bin/velero
      $ velero version
      Client:
        Version: v1.2.0
        Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
      Server:
        Version:

    2. 编辑credentials-velero文件, 设置Minio服务的访问权限信息。
      [default]
      aws_access_key_id=<YourAccessKeyId>
      aws_secret_access_key=<YourAccessKeySecret>
      说明 此处的YourAccessKeyIdYourAccessKeySecret为访问Minio服务的登录用户名和密码。

    3. 替换MINIO_SERVER_ADDRESS的值,并使用以下命令部署Velero服务。
      velero install --provider aws --image registry.cn-hangzhou.aliyuncs.com/acs/velero:v1.2.0 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://MINIO_SERVER_ADDRESS:9000 --use-restic --plugins registry.cn-hangzhou.aliyuncs.com/acs/velero-plugin-for-aws:v1.0.0 --wait

    4. 查看pod的运行状态。
      kubectl -n velero get po
      NAME                      READY   STATUS    RESTARTS   AGE
      restic-fqwsc              1/1     Running   0          41s
      restic-kfzqt              1/1     Running   0          41s
      restic-klxhc              1/1     Running   0          41s
      restic-ql2kr              1/1     Running   0          41s
      restic-qrsrn              1/1     Running   0          41s
      restic-srjmm              1/1     Running   0          41s
      velero-67b975f5cb-68nj4   1/1     Running   0          41s

  3. 备份wordpress应用。
    • 如果只需要备份wordpress应用而不备份PV数据,执行以下操作。
      $ velero backup create wordpress-backup-without-pv --include-namespaces wordpress
      Backup request "wordpress-backup-without-pv" submitted successfully.
      Run `velero backup describe wordpress-backup-without-pv` or `velero backup logs wordpress-backup-without-pv` for more details.
      
      $ velero backup get
      NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
      wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
    • 如果需要备份带PV数据的wordpress应用,执行以下操作。
      # 首先需要为挂载pv数据卷的pod添加annotation, 例如wordpress应用运行了2个pod, 分别为wordpress-7cf5849f47-mbvx4 mysql-74dddbdcc8-h2tls, wordpress-7cf5849f47-mbvx4
      # 挂载的volume名为mysql-persistent-storage, mysql-74dddbdcc8-h2tls挂载的volume名为wordpress-persistent-storage, 则添加annotation的命令为
      $ kubectl -n wordpress annotate pod/wordpress-7cf5849f47-mbvx4 backup.velero.io/backup-volumes=wordpress-persistent-storage
      pod/wordpress-7cf5849f47-mbvx4 annotated
      $ kubectl -n wordpress annotate pod/mysql-74dddbdcc8-h2tls backup.velero.io/backup-volumes=mysql-persistent-storage
      pod/mysql-74dddbdcc8-h2tls annotated
      
      # 备份wordpress
      $ velero backup create wordpress-backup-with-pv --include-namespaces wordpress
      Backup request "wordpress-backup-with-pv" submitted successfully.
      Run `velero backup describe wordpress-backup-with-pv` or `velero backup logs wordpress-backup-with-pv` for more details.
      $ velero backup get
      NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
      wordpress-backup-with-pv      Completed   2019-12-12 14:23:40 +0800 CST   29d       default            <none>
      wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
    查看Minio服务器可以看到备份的文件。Kubernetes应用迁移教程

步骤四:在ACK上部署Velero并恢复应用wordpress

  1. 在ACK部署Velero。
    操作步骤同自建Kubernetes集群部署Velero。

    部署完成后,查看backup列表的运行状态。

    $ velero backup get
    NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
    wordpress-backup-with-pv      Completed   2019-12-12 14:23:40 +0800 CST   29d       default            <none>
    wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>

  2. 在ACK中恢复wordpress应用。
    wordpress应用使用NFS类型持久化数据卷,相应的,在ACK中我们可以适配NAS volume,在本示例中,我们创建与Wordpress应用所使用的StorageClass
    NFS, 但后端存储介质使用SSD云盘块存储。

    本示例使用ACK集群使用CSI plugin, 请参见动态云盘卷。

    1. 创建StorageClass。
      如果在ACK上备份wordpress应用不带pv数据,则跳过此步骤。

      $ cat nfs.yaml
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         name: nfs
      provisioner: diskplugin.csi.alibabacloud.com
      parameters:
          type: cloud_ssd
      reclaimPolicy: Retain
      
      $ kubectl apply -f nfs.yaml
      storageclass.storage.k8s.io/nfs created

    2. 恢复wordpress应用。
      $ velero restore create --from-backup wordpress-backup-with-pv
      $ velero restore get
      NAME                                      BACKUP                     STATUS       WARNINGS   ERRORS   CREATED                         SELECTOR
      wordpress-backup-with-pv-20191212152745   wordpress-backup-with-pv   InProgress   0          0        2019-12-12 15:27:45 +0800 CST   <none>
      $ velero restore get
      此时查看wordpress应用运行情况,会有镜像拉取失败的问题。

      $ kubectl -n wordpress get po
      NAME                         READY   STATUS         RESTARTS   AGE
      mysql-669b4666cd-trsnz       0/1     ErrImagePull   0          19m
      mysql-74dddbdcc8-h2tls       0/1     Init:0/1       0          19m
      wordpress-7cf5849f47-mbvx4   0/1     Init:0/1       0          19m
      wordpress-bb5d74d95-xcjxw    0/1     ErrImagePull   0          19m

    3. 编辑Deployment把image字段替换成步骤一:容器镜像迁移上云中迁移后的镜像地址。
      # edit 编辑deployment并修改image url
      $ kubectl -n wordpress edit deployment mysql
      $ kubectl -n wordpress edit deployment wordpress

步骤五:验证wordpress应用服务

查看wordpress应用运行情况。

$ kubectl -n wordpress get po
NAME                         READY   STATUS    RESTARTS   AGE
mysql-678b5d8499-vckfd       1/1     Running   0          100s
wordpress-8566f5f7d8-7shk6   1/1     Running   0          3m18s

测试环境绑定hosts后,通过http://wordpress.myk8s.paas.com:31570/访问wordpress应用。Kubernetes应用迁移教程