This document describes the specifications for designing the Binary Provisioning Agent Custom resource Definition required for the Integrated Cloud Native Akraino project iteratively.
1.1. Iterations
This task will evolve over time and we will keep iterating and modifying this document. Iterations are small steps that will happen within a short period of time with specific tasks to roll out POCs as quickly as possible for feedback inorder to adjust the focus quickly.
The BPA is part of the infra local controller which runs as a bootstrap k8s cluster in the ICN project. As described in Integrated Cloud Native Akraino project, the purpose of the BPA is to install packages that cannot be installed using kubectl. It will be called once the operating system (Linux) has been installed in the compute nodes by the baremetal operator. The Binary Provisioning Agent will carry out the following functions;
For more information on the BPA functions, check out the ICN Akraino project link above
We do not intend to make any changes to the existing kubernetes API in order to implement the specifications described in this document. We will simply be extending the Kubernetes API using Custom Resource Definition as described here and then creating a custom controller that will handle the requirements of our provisioning Agent custom resource.
Prerequisites: This workflow assumes that the baremetal CR and baremetal operator have been created and has successfully installed the compute nodes with Linux OS. It also assumes that the BPA controller is running.
Fig 1: Illustration of the proposed workflow
`
Workflow Summary/Description
The BPA CRD tells the Kubernetes API how to expose the provisioning custom resource object. The CRD yaml file is applied using
“kubectl create -f bpa_v1alpha1_provisioning_crd.yaml” See below for the CRD definition.
5.1. BPA CRD Yaml File (*_crd.yaml)
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: provisionings.bpa.akraino.org spec: group: bpa.akraino.org names: kind: Provisioning listKind: ProvisioningList plural: provisionings singular: provisioning shortNames: - bpa scope: Namespaced subresources: status: {} validation: openAPIV3Schema: properties: apiVersion: description: type: string kind: description: type: string metadata: type: object spec: type: object status: type: object version: v1alpha1 versions: - name: v1alpha1 served: true storage: true |
5.2. Provisioning Agent Object Definition( *_types,go)
The provisioning_types.go file is the API for the provisioning agent custom resource.
// ProvisioningSpec defines the desired state of Provisioning type ProvisioningSpec struct { Masters []Master `json:"master,omitempty"` Workers []Worker `json:"worker,omitempty"` Replicas int32 `json:"replicas,omitempty"` } // ProvisioningStatus defines the observed state of // Provisioning type ProvisioningStatus struct { // Names of provisioning agent pods when a deployment // is running PodAgents []string `json"podAgents,omitempty"` } // Provisioning is the Schema for the provisionings API type Provisioning struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec ProvisioningSpec `json:"spec,omitempty"` Status ProvisioningStatus `json:"status,omitempty"` } // ProvisioningList contains a list of Provisioning type ProvisioningList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` Items []Provisioning `json:"items"` } // master struct contains resource requirements for a master // node type Master struct { CPU int32 `json:"cpu,omitempty"` Memory string `json:"memory,omitempty"` } // worker struct contains resource requirements for a worker node type Worker struct { CPU int32 `json:"cpu,omitempty"` Memory string `json:"memory,omitempty"` SRIOV bool `json:"sriov,omitempty"` QAT bool `json:"qat,omitempty"` } |
The variables in the ProvisioningSpec struct are used to create the data structures in the yaml spec for the custom resource. Three variables are added to the ProvisioningSpec struct;
In the ProvisioningStatus struct, the podAgents variable is defined. This variable will display a list of the names of the pods that are part of the provisioning agent deployment when the status of the deployment is queried.
5.3. Sample Provisioning CR YAML file
apiVersion: bpa.akraino.org/v1alpha1 kind: Provisioning metadata: name: provisioning-sample spec: master: cpu: 10 memory: 4Gi worker: cpu: 20 memory: 8Gi replicas: 2 |
The YAML file above can be used to create a provisioning custom resource which is an instance of the provisioning CRD describes above. The spec.master field corresponds to the Masters variable in the ProvisioningSpec struct of the *-types.go file, while the spec.worker field corresponds to the Workers variable in the ProvisioningSpec struct of the *-types.go file and the spec.replica field corresponds to the Replicas variable in the same struct.
Based on the values above, when the BPA operator gets the baremetal hosts object (Step 5in figure 1), it would assign hosts with 10 CPUs and 4Gi memory the role of master and it would assign hosts with 20CPUs and 8Gi memory the role of worker.
The replicas spec tells the controller that in this CR deployment, at any instance, there should be 2 pods running the provisioning agent. If a pod dies, the controller reconciles this and immediately starts a new pod running the provisioning agent.
This proposal would make it possible to assign roles to nodes based on the features discovered. Currently, the proposal makes use of CPU, memory, SRIOV and QAT. However the baremetal operator list returns much more information about the nodes, we would be able to extend this feature to allow the operator assign roles based on more complex requirements such as CPU model. This would feed into Hardware Platform Awareness (HPA)