...
- Provisioning controller (PC) Micro Services
- Binary Provisioning Manager (BPM) Micro services
- K8s Provisioning Manager (KPM) Micro-services
- Certificate and Secret Management (CSM) related Micro-services
- Cluster-API related Micro-services
- MongoDB for storing packages and OS images.Prometheus: Monitoring and alerting
Since we expect the infra-global-controller to be reachable from the Internet, we should be secured using
...
Local Controller: kubeadm, Metal3, Bare Metal Operator, Ironic, Prometheus, EMCO
Global Controller: kubeadm, KUD, K8s Provisioning Manager, Binary Provisioning Manager, Prometheus, CSM
R5 Release cover only Infra local controller:
...
One of the major challenges to cloud admin managing multiple clusters in different edge location is coordinate control plane of each cluster configuration remotely, managing patches and updates/upgrades across multiple machines. Cluster-API provides declarative APIs to represent clusters and machines inside a cluster. Cluster-API provides the abstraction for various common logic that can be seen in various cluster provider such as GKE, AWS, Vsphere. Cluster-API consolidated all those logic provide abstractions for all those logic functions such as grouping machines for the upgrade, auto-scaling mechanism.
In ICN family stack, Bare Metal Operator from Metal3 project is used as bare metal provider. It is used as a machine actuator that uses Ironic to provide K8s API to manage the physical servers that also run K8s clusters on bare-metal host.
KuD
K8s deployment (KUD) is a project that uses Kubespray to bring up a K8s deployment and some add-ons on a provisioned machine. One of the K8s clusters with high availability, which is provisioned and configured by KUD, will be used to deploy EMCO on K8s. ICN family uses Edge Multi-Cluster Orchestration for service orchestration. EMCO provides a set of helm chart to be used to run the workloads on a Multi - cluster.
EMCO Block and Modules:
EMCO will be the Service Orchestration Engine in ICN family and is responsible for the VNF life cycle management, tenant management and Tenant resource quota allocation and managing Resource Orchestration engine (ROE) to schedule VNF workloads with Multi-site scheduler awareness and Hardware Platform abstraction (HPA). It can be used to deploy the K8s App components (as shown in fig. II), NFV Specific components and NFVi SDN controller in the edge cluster. In R5 release EMCO will be used to deploy the K8s add-on such as Virtlet, OVN, NFD, and Intel device plugins such as SRIOV in the edge location (as shown in figure I). Required an Akraino dashboard that sits on the top of EMCO to deploy the VNFs.
K8s Block and Modules:
K8s will be the Resource Orchestration Engine in ICN family to manage Network, Storage and Compute resource for the VNF application. ICN family will be using multiple container runtimes as Virtlet and docker as a de-facto container runtime. Each release supports different container runtimes that are focused on use cases.
K8s module is divided into 3 groups - K8s App components, NFV specific components and NFVi SDN controller components, all these components will be installed using EMCO
K8s App components: This block has K8s storage plugins, container runtime, OVN for networking, Service proxy and Prometheus for monitoring, and responsible application management
NFV Specific components: This block is responsible for K8s compute management to support both software and hardware acceleration (including network acceleration) with CPU pinning and Device plugins such as SRIOV
SDN Controller components: This block is responsible for managing SDN controller and to provide additional features such as Service Function chaining (SFC) and Network Route manager.
Modules Design & Architecture:
Metal3:
ICN uses Metal3 project for provisioning server in the edge locations, ICN project uses IPMI protocol to identify the servers in the edge locations, and use Ironic & Ironic - Inspector to provision the OS in the edge location. For R5 release, ICN project provision Ubuntu 18.04 in each server, and uses the distinguished network such provisioning network and bare-metal network for inspection and IPMI provisioning.
ICN project injects the user data in each server regarding network configuration, grub update to enable IOMMU, remote command execution using ssh and maintain a common secure mechanism for all provisioning the servers. Each local controller maintains IP address management for that edge location. For more information refer - Metal3 Bare Metal Operator in ICN stack
BPA Operator:
ICN uses the BPA operator to install KUD. It can install KUD either on baremetal hosts or on Virtual Machines. The BPA operator is also used to install software on the machines after KUD has been installed successfully
KUD Installation
Baremetal Hosts: When a new provisioning CR is created, the BPA operator function is triggered, it then uses a dynamic client to get a list of all baremetal hosts that were provisioned using Metal3. It reads the MAC addresses from the provisioning CR and compares with the baremetal hosts list to confirm that a host with that MAC address exists. If it exists, it then searches the DHCP lease file for corresponding IP address of the host, using the IP addresses of all the hosts in the provisioning CR, it then creates an inventory file and triggers a job that installs KUD on the machines using the inventory file. When the job is completed successfully, a K8s cluster is running in the baremetal hosts. The BPA operator then creates a ConfigMap using the hosts name as keys and their corresponding IP addresses as values. If a host containing a specified MAC address does not exist, the BPA operator throws an error.
Virtual Machines : ICN project uses Virtlet for provisioning virtual machines in the edge locations. For this release, it involves a nested K8s implementation. K8s is first installed with Virtlet. Pod spec files are created with cloud init user data, network annotation with mac address, CPU and Memory requests. Virtlet VMs are created as per cluster spec or requirement. Corresponding provisioning custom resources are created to match the mac addresses of the Virtlet VMs.
BPA operator checks the provisioning custom resource and maps the mac address(es) to the running Virtlet VM(s). BPA operator gets the IP addresses of those VMs and initiates an installer job which runs KUD scripts in those VMs. Upon completion, the K8s cluster is ready running in the Virtlet VMs.
Software Installation
When a new software CR is created, the reconcile loop is triggered, on seeing that it is a software CR, the BPA operator checks for a ConfigMap with a cluster label corresponding to that in the software CR, if it finds one, it gets the IP addresses of all the master and worker nodes, ssh's into the hosts and installs the required software. If no corresponding config map is found, it throws an error.
Refer
BPA Rest Agent:
Provides a straightforward RESTful API that exposes resources: Binary Images, Container Images, and OS Images. This is accomplished by using MinIO for object storage and MongoDB for metadata.
POST - Creates a new image resource using a JSON file.
GET - Lists available image resources.
PATCH - Uploads images to the MinIO backend and updates MongoDB.
DELETE - Removes the image from MinIO and MongoDB
More on BPA Restful API can be found at ICN Rest API.
EMCO:
EMCO is used as Service orchestration in ICN BP. ICN BP developed containerized KUD multi-cluster to install the EMCO as a plugin in any cluster provisioned by BPA operator. EMCO installed Composite vFW application to install in any edge location.
SDEWAN:
SDEWAN CNF module is worked as a software-defined router located in each edge location and central hub K8s cluster to manage central-edge and edge-edge communication. It's functionality is realized via CNF (Containerized Network Function) and deployed by K8s, it is based on OpenWRT (an open-source project based on Linux, and used on embedded devices to route network traffic) and leverages Linux kernel functionality for packet processing to support network functionalities such as multiple wan link support (mwan3), firewall/SNAT/DNAT (fw3) , IPSec (strongswan) etc. It exposes Restful APIs for configuration, detail information can be found at: SDEWAN CNF
SDEWAN Configure Agent (also named SDEWAN Controller) module is worked as K8s controller located in each edge location and central hub K8s cluster to support configuration of SDEWAN CNF functionalities (e.g. mwan3, firwall, SNAT, DNAT, IPSec etc.) and monitor SDEWAN CNF status. It exposes CRDs to support configuration via K8s API server for unified authentication and authorization, detail information can be found at: SDEWAN CRD Controller
Cloud Storage:
Cloud Storage (Cloud Storage Design) act as storage service and plugins, currently can divide into two parts:
- Storage Service for Local controller: which used by BPA Rest Agent to provide storage service for image objects with binary, container and operating system. There are 2 solutions, MinIO and GridFS, with the consideration of Cloud native and Data reliability, we propose to use MinIO, which is CNCF project for object storage and compatible with Amazon S3 API, and provide language plugins for client application, it is also easy to deploy in K8s and flexible scale-out. MinIO also provide storage service for HTTP Server. Since MinIO need export volume in bootstrap, local-storage is a simple solution but lack of reliability for the data safety, we will switch to reliability volume provided by Ceph CSI RBD in next release.
- Optane Persistent Memory plugin in KUD, which can provide LVM and direct volumes on Optane PM namespaces, since the Optane PM has high performance and low latency compared with normal SSD storage device, it can be used as cache, metadata volume or other high throughput and low latency scenarios.
Software components:
...
Components
...
Link
...
Akraino Release target
...
Provision stack - Metal3
...
https://github.com/akraino-icn/baremetal-operator - v2.0-icn
...
R5
...
Host Operating system
...
Ubuntu 18.04
...
R5
...
NIC drivers
...
...
R5
...
Intel® Optane™ DC 256GB Persistent Memory Module -
PMDK: Persistent Memory Development Kit - https://github.com/pmem/pmdk/
...
EMCO
(formerly known as ONAP4K8s)
...
https://github.com/open-ness/EMCO
...
R5
...
SDEWAN CNFs
...
https://github.com/akraino-edge-stack/icn-sdwan - v1.0
https://hub.docker.com/repository/docker/integratedcloudnative/openwrt - 0.3.1
...
R5
...
KUD
...
https://git.onap.org/multicloud/K8s/
...
R5
...
Kubespray
...
https://github.com/kubernetes-sigs/kubespray v2.14.1
...
R5
...
K8s
...
https://github.com/kubernetes/kubeadm - v1.18.9
...
R5
...
Docker
...
https://github.com/docker - 19.03
...
R5
...
Virtlet
...
https://github.com/Mirantis/virtlet -1.4.4
...
R5
...
SDN - OVN
...
https://github.com/akraino-icn/ovn/ - v20.06.0
(mirror repo - https://github.com/ovn-org/ovn)
...
R5
...
https://github.com/akraino-icn/ovs - v2.14.0
(mirror repo - https://github.com/openvswitch/ovs )
...
Ansible
...
https://github.com/ansible/ansible - 2.9.7
...
R5
...
Helm
...
https://github.com/helm/helm - 3.2.4
...
R5
...
Istio
...
https://github.com/istio/istio - 1.0.3
...
R5
...
Rook/Ceph
...
https://rook.io/docs/rook/v1.0/helm-operator.html v1.0
...
R5
...
MetalLB
...
https://github.com/danderson/metallb/releases - v0.7.3
...
R5
...
https://github.com/akraino-edge-stack/icn-sdwan - v1.0
https://hub.docker.com/repository/docker/integratedcloudnative/sdewan-controller - 0.3.0
...
Node Feature Discovery
...
https://github.com/kubernetes-sigs/node-feature-discovery - v0.7.0
...
R5
...
CNI
...
https://github.com/coreos/flannel/ - v0.12.0
https://github.com/containernetworking/cni - release tag v0.7.0
https://github.com/containernetworking/plugins - v0.8.7
https://github.com/akraino-icn/multus-cni - v3.7
https://github.com/k8snetworkplumbingwg/sriov-cni
...
R5
...
. In ICN family stack, Bare Metal Operator from Metal3 project is used as bare metal provider. It is used as a machine actuator that uses Ironic to provide K8s API to manage the physical servers that also run K8s clusters on bare-metal host.
KuD
K8s deployment (KUD) is a project that uses Kubespray to bring up a K8s deployment and some add-ons on a provisioned machine. One of the K8s clusters with high availability, which is provisioned and configured by KUD, will be used to deploy EMCO on K8s. ICN family uses Edge Multi-Cluster Orchestration for service orchestration. EMCO provides a set of helm chart to be used to run the workloads on a multi-cluster.
EMCO Block and Modules:
EMCO will be the Service Orchestration Engine in ICN family and is responsible for the VNF life cycle management, tenant management and Tenant resource quota allocation and managing Resource Orchestration engine (ROE) to schedule VNF workloads with Multi-site scheduler awareness and Hardware Platform abstraction (HPA). It can be used to deploy the K8s App components (as shown in fig. II), NFV Specific components and NFVi SDN controller in the edge cluster. In R5 release EMCO will be used to deploy the K8s add-on such as OVN, NFD, and Intel device plugins such as SRIOV in the edge location (as shown in figure I). Required an Akraino dashboard that sits on the top of EMCO to deploy the VNFs.
K8s Block and Modules:
K8s will be the Resource Orchestration Engine in ICN family to manage Network, Storage and Compute resource for the VNF application. ICN family will be using docker as a de-facto container runtime. Each release supports different container runtimes that are focused on use cases.
K8s module is divided into 3 groups - K8s App components, NFV specific components and NFVi SDN controller components, all these components will be installed using EMCO
K8s App components: This block has K8s storage plugins, container runtime, OVN for networking, Service proxy, and responsible application management
NFV Specific components: This block is responsible for K8s compute management to support both software and hardware acceleration (including network acceleration) with CPU pinning and Device plugins such as SRIOV
SDN Controller components: This block is responsible for managing SDN controller and to provide additional features such as Service Function chaining (SFC) and Network Route manager.
Modules Design & Architecture:
Metal3:
ICN uses Metal3 project for provisioning server in the edge locations, ICN project uses IPMI protocol to identify the servers in the edge locations, and use Ironic & Ironic - Inspector to provision the OS in the edge location. For R5 release, ICN project provision Ubuntu 18.04 in each server, and uses the distinguished network such provisioning network and bare-metal network for inspection and IPMI provisioning.
ICN project injects the user data in each server regarding network configuration, grub update to enable IOMMU, remote command execution using ssh and maintain a common secure mechanism for all provisioning the servers. Each local controller maintains IP address management for that edge location. For more information refer - Metal3 Bare Metal Operator in ICN stack
BPA Operator:
ICN uses the BPA operator to install KUD. It can install KUD either on baremetal hosts or on Virtual Machines. The BPA operator is also used to install software on the machines after KUD has been installed successfully
KUD Installation
Baremetal Hosts: When a new provisioning CR is created, the BPA operator function is triggered, it then uses a dynamic client to get a list of all baremetal hosts that were provisioned using Metal3. It reads the MAC addresses from the provisioning CR and compares with the baremetal hosts list to confirm that a host with that MAC address exists. If it exists, it then searches the DHCP lease file for corresponding IP address of the host, using the IP addresses of all the hosts in the provisioning CR, it then creates an inventory file and triggers a job that installs KUD on the machines using the inventory file. When the job is completed successfully, a K8s cluster is running in the baremetal hosts. The BPA operator then creates a ConfigMap using the hosts name as keys and their corresponding IP addresses as values. If a host containing a specified MAC address does not exist, the BPA operator throws an error.
Software Installation
When a new software CR is created, the reconcile loop is triggered, on seeing that it is a software CR, the BPA operator checks for a ConfigMap with a cluster label corresponding to that in the software CR, if it finds one, it gets the IP addresses of all the master and worker nodes, ssh's into the hosts and installs the required software. If no corresponding config map is found, it throws an error.
Refer
BPA Rest Agent:
Provides a straightforward RESTful API that exposes resources: Binary Images, Container Images, and OS Images. This is accomplished by using MinIO for object storage and MongoDB for metadata.
POST - Creates a new image resource using a JSON file.
GET - Lists available image resources.
PATCH - Uploads images to the MinIO backend and updates MongoDB.
DELETE - Removes the image from MinIO and MongoDB
More on BPA Restful API can be found at ICN Rest API.
EMCO:
EMCO is used as Service orchestration in ICN BP. ICN BP developed containerized KUD multi-cluster to install the EMCO as a plugin in any cluster provisioned by BPA operator. EMCO installed Composite vFW application to install in any edge location.
SDEWAN:
SDEWAN CNF module is worked as a software-defined router located in each edge location and central hub K8s cluster to manage central-edge and edge-edge communication. It's functionality is realized via CNF (Containerized Network Function) and deployed by K8s, it is based on OpenWRT (an open-source project based on Linux, and used on embedded devices to route network traffic) and leverages Linux kernel functionality for packet processing to support network functionalities such as multiple wan link support (mwan3), firewall/SNAT/DNAT (fw3) , IPSec (strongswan) etc. It exposes Restful APIs for configuration, detail information can be found at: SDEWAN CNF
SDEWAN Configure Agent (also named SDEWAN Controller) module is worked as K8s controller located in each edge location and central hub K8s cluster to support configuration of SDEWAN CNF functionalities (e.g. mwan3, firwall, SNAT, DNAT, IPSec etc.) and monitor SDEWAN CNF status. It exposes CRDs to support configuration via K8s API server for unified authentication and authorization, detail information can be found at: SDEWAN CRD Controller
Cloud Storage:
Cloud Storage (Cloud Storage Design) act as storage service and plugins, currently can divide into two parts:
- Storage Service for Local controller: which used by BPA Rest Agent to provide storage service for image objects with binary, container and operating system. There are 2 solutions, MinIO and GridFS, with the consideration of Cloud native and Data reliability, we propose to use MinIO, which is CNCF project for object storage and compatible with Amazon S3 API, and provide language plugins for client application, it is also easy to deploy in K8s and flexible scale-out. MinIO also provide storage service for HTTP Server. Since MinIO need export volume in bootstrap, local-storage is a simple solution but lack of reliability for the data safety, we will switch to reliability volume provided by Ceph CSI RBD in next release.
- Optane Persistent Memory plugin in KUD, which can provide LVM and direct volumes on Optane PM namespaces, since the Optane PM has high performance and low latency compared with normal SSD storage device, it can be used as cache, metadata volume or other high throughput and low latency scenarios.
Software components:
Please refer to list of software components in the ICN R5 Release Notes
...
Hardware and Software Management
...