Storage Service for Local Controller
The requirement initial from RESTful API Binary Provisioning Agent which provide interface to operator to upload image objects.
There are 3 key functionality for RESTful API Agent:
- Image provisioning (upload)
- REST API (POST/PUT/PATCH/DELETE etc)
- Support resumable upload
Backend Storage Solutions
There are several considerations for the backend storage solution:
- Data reliability: some mechanism like replication to make sure the data can be recovered.
- Cloud native: Support docker or Kubernetes deployment.
- Amazon S3 API compatible: Support S3 API which is factual standard.
- Multipart upload is better: which can speed up or resume when upload thread is break.
Implementation
We choose MinIO as the backend storage solution which provide Cloud Storage Service for Local Controller.
The implementation for the module is to develop MinIO client plugin based on MinIO Golang API, and integrate with REST API agent, include interface for MinIO client plugin as:
- Initialize(), which create new client and setup buckets
- PutImage(), put the object file to MinIO server, support /PUT
- PatchImage(), Put the object with offset and compose, support /PATCH
- DeleteImage(), Delete the image
- CleanupImages(), Cleanup and remove incomplete upload threads
Deployment
the reliable volume is provided by Ceph cluster with CSI, and which used by MinIO standalone server, MinIO clinet plugin code interact with S3 API to provide object service for REST API agent.
Optane PM (Persistent Memory) Plugin
Optane PM and PMEM-CSI
The term persistent memory is used to describe technologies which allow programs to access data as memory, directly byte-addressable, while the contents are non-volatile, preserved across power cycles. It has aspects that are like memory, and aspects that are like storage.
Intel Optane DC Persistent Memory is an innovative technology that delivers a unique combination of affordable large memory capacity and persistence (non-volatility). The persistent memory technology can help boost the performance of data-intensive applications, such as in-memory analytics, databases, content delivery networks, and high performance computing (HPC), as well as deliver consistent service levels at scale with higher virtual machine and container density.
To enable the Optane PM (Persistent Memory) on cloud native application, Intel has develop the PMEM-CSI driver for Kubernetes. Intel PMEM-CSI is a CSI (Container Storage Interface) storage driver for container orchestrators like Kubernetes. It makes local persistent memory (PMEM) available as a filesystem volume to container applications. It can currently utilize non-volatile memory devices that can be controlled via the libndctl utility library.
The PMEM-CSI driver can operate in two different device modes: LVM and direct.
- In Logical Volume Management (LVM) mode the PMEM-CSI driver uses LVM for logical volume Management to avoid the risk of fragmentation. The LVM logical volumes are served to satisfy API requests. There is one volume group created per region, ensuring the region-affinity of served volumes.
- In direct device mode PMEM-CSI driver allocates namespaces directly from the storage device. This creates device space fragmentation risk, but reduces complexity and run-time overhead by avoiding additional device mapping layer. Direct mode also ensures the region-affinity of served volumes, because provisioned volume can belong to one region only.
For the Optane PM plugin in ICN, we support LVM mode, which provide CSI plugin and CSI driver, the framework as following chart.
The common usage scenario for Optane PM is mount as cache partition or store metadata for filesystem or clusters (Ceph for example), this work will be design in future release.
Implementation
Optane PM plugin is part of KUD plugin addons, the bring up process defined in KUD deployment_infra ansible playbook.
Optane PM CSI plugin and driver layout as following chart, we create example application with StorageClass and PVC for dynamic volume provisioning.
