云实验室(24) - kubesphere使用ceph-csi进行存储

1. 准备工作

由于虚拟机没有科学上网,一些k8s的镜像拉取不到,固使用一些其他源代替,顺次执行以下命令在各个节点上(注意不要一次全部执行,要一条一条来,确保镜像全部拉取成功)
注意,以下仓库地址可能会变化,需要自己找到拉取不到的镜像,检索一下可用的镜像源自行替换,目前基于csi 的helm版本是3.7.1的.

docker pull registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.5.1
docker tag registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.5.1 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1

docker pull registry.aliyuncs.com/google_containers/csi-attacher:v3.5.0
docker tag registry.aliyuncs.com/google_containers/csi-attacher:v3.5.0 registry.k8s.io/sig-storage/csi-attacher:v3.5.0

docker pull registry.aliyuncs.com/google_containers/csi-snapshotter:v6.0.1
docker tag registry.aliyuncs.com/google_containers/csi-snapshotter:v6.0.1 registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1

docker pull registry.aliyuncs.com/it00021hot/csi-resizer:v1.5.0
docker tag registry.aliyuncs.com/it00021hot/csi-resizer:v1.5.0 registry.k8s.io/sig-storage/csi-resizer:v1.5.0  
docker pull registry.aliyuncs.com/google_containers/csi-provisioner:v3.2.1 
docker tag registry.aliyuncs.com/google_containers/csi-provisioner:v3.2.1  gcr.io/k8s-staging-sig-storage/csi-provisioner:v3.2.1

2. 创建k8s的存储卷类型和csi适配中间层资源

参考文档

https://kubesphere.com.cn/docs/installing-on-linux/persistent-storage-configurations/install-ceph-csi-rbd/

新建如下文件

ceph-csi-rbd-sc.yaml

apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: kube-system
stringData:
  userID: admin
  userKey: "AQD354hhgYQCDhAA86PVHdMHrdGyTLAKbww3oQ=="    # <--ToBeReplaced-->
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
   annotations:
     storageclass.beta.kubernetes.io/is-default-class: "true"
     storageclass.kubesphere.io/supported-access-modes: '["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]'
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: "b487ad26-3e82-44e9-b3ee-85f92b6a30a3"   # <--ToBeReplaced-->
   pool: "kube-pool"
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: kube-system
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: kube-system
   csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard

ceph-csi-rbd.yaml

csiConfig:
  - clusterID: "b487ad26-3e82-44e9-b3ee-85f92b6a30a3"  # <--ToBeReplaced-->
    monitors:
      - "192.168.0.108:6789"

3. kk 加入插件配置

config-sample.yaml 局部 :

spec:
  hosts:
  - {name: node1, address: 192.168.0.123, internalAddress: 192.168.0.123, privateKeyPath: "~/.ssh/id_rsa"}
  - {name: node2, address: 192.168.0.124, internalAddress: 192.168.0.124, privateKeyPath: "~/.ssh/id_rsa"}
  - {name: node3, address: 192.168.0.125, internalAddress: 192.168.0.125, privateKeyPath: "~/.ssh/id_rsa"}
  roleGroups:
    etcd:
    - node1
    master:
    - node1
    worker:
    - node1
    - node2
    - node3
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.19.8
    imageRepo: kubesphere
    clusterName: cluster.local
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
  registry:
    registryMirrors: []
    insecureRegistries: []
  addons:
  - name: ceph-csi-rbd
    namespace: kube-system
    sources:
      chart:
        name: ceph-csi-rbd
        repo: https://ceph.github.io/csi-charts
        valuesFile: /root/ceph-csi-rbd.yaml
  - name: ceph-csi-rbd-sc
    sources:
      yaml:
        path:
        - /root/ceph-csi-rbd-sc.yaml

全文 : config-sample.yaml

主要关注 addons部分,valuesFile,path要和刚才创建的yaml地址匹配

3. 安装插件

./kk create cluster -f config-sample.yaml

效果