Introduce

This plugin describe the deployment with containerized Ceph cluster by Rook operator and provide storage service in edge location node.

As a distributed backend storage, Ceph is widely used on Cloud and Edge solutions, it can provide Block, Filesystem and Object storage service. Traditional deployment for Ceph tools include: ceph-deploy, puppet, ansible etc.

Rook is a storage orchestrator for Kubernetes, which turns distributed storage system into self-managing, self-scaling, self-healing storage services, compared with traditional tools, there are several expectations by using Rook:

  • Reduced deployment time for new clusters
  • Simplified upgrades
  • More agile horizontal scaling
  • Better failure tolerance
  • Reduced reliance on expert Ceph operators

CSI (Container Storage Interface) is a standard for exposing arbitrary block and file storage storage system to containerized workload on Container Orchestration Systems (COs) like Kubernetes. Also Ceph-CSI is supported by Rook from v1.0.


Implement

Rook implementation include two parts, Rook operator and Rook Ceph cluster.

Rook operator include CRDs and also including Ceph-CSI support for volume provisioning.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-operator
  namespace: rook-ceph
labels:
  operator: rook
  storage-backend: ceph
spec:
  selector:
    matchLabels:
      app: rook-ceph-operator
  replicas: 1
  template:
    metadata:
      labels:
        app: rook-ceph-operator
    spec:
      serviceAccountName: rook-ceph-system
      containers:
      - name: rook-ceph-operator
        image: rook/ceph:v1.0.4
        args: ["ceph", "operator"]
        volumeMounts:
        - mountPath: /var/lib/rook
          name: rook-config
        - mountPath: /etc/ceph
          name: default-config-dir
         env:
         - name: ROOK_CURRENT_NAMESPACE_ONLY
           value: "true"
         # CSI enablement
        - name: ROOK_CSI_ENABLE_CEPHFS
          value: "true"
        - name: ROOK_CSI_CEPHFS_IMAGE
          value: "quay.io/cephcsi/cephfsplugin:v1.0.0"
        - name: ROOK_CSI_ENABLE_RBD
          value: "true"
        - name: ROOK_CSI_RBD_IMAGE
          value: "quay.io/cephcsi/rbdplugin:v1.0.0"
        - name: ROOK_CSI_REGISTRAR_IMAGE
          value: "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
        - name: ROOK_CSI_PROVISIONER_IMAGE
          value: "quay.io/k8scsi/csi-provisioner:v1.0.1"
        - name: ROOK_CSI_SNAPSHOTTER_IMAGE
          value: "quay.io/k8scsi/csi-snapshotter:v1.0.1"
        - name: ROOK_CSI_ATTACHER_IMAGE
          value: "quay.io/k8scsi/csi-attacher:v1.0.1"
         # The name of the node to pass with the downward API
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
          # The pod name to pass with the downward API
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
          # The pod namespace to pass with the downward API
       - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumes:
       - name: rook-config
          emptyDir: {}
      - name: default-config-dir
         emptyDir: {}

Rook Ceph cluster include settings for Ceph monitor and storage, currently the setting as following:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: ceph/ceph:v13.2.2-20190410
allowUnsupported: false
dataDirHostPath: /var/lib/rook
mon:
count: 3
allowMultiplePerNode: true
dashboard:
enabled: true
network:
hostNetwork: false
rbdMirroring:
workers: 0
annotations:
resources:
storage: # cluster level storage configuration and selection
useAllNodes: true
useAllDevices: false
deviceFilter:
location:
config:
storeType: filestore
metadataDevice: # "md0" specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
databaseSizeMB: "10240" # uncomment if the disks are smaller than 100 GB
journalSizeMB: "10240" # uncomment if the disks are 20 GB or smaller
directories:
- path: "/var/lib/rook/storage-dir"
  • No labels