GoalsThis document describes the specifications for designing the Binary Provisioning Agent Custom resource Definition required for the Integrated Cloud Native Akraino project iteratively.

Iterations

This task will evolve over time and we will keep iterating and modifying this document. Iterations are small steps that will happen within a short period of time with specific tasks to roll out POCs as quickly as possible for feedback inorder to adjust the focus quickly.

Overview of BPA

The BPA is part of the infra local controller which runs as a bootstrap k8s cluster in the ICN project. As described in Integrated Cloud Native Akraino project, the purpose of the BPA is to install packages that cannot be installed using kubectl. It will be called once the operating system (Linux) has been installed in the compute nodes by the baremetal operator. The Binary Provisioning Agent will carry out the following functions;

1. Get site specific information containing information about compute nodes and their resource capabilities.
2. Assign roles to compute hosts and create the hosts.ini file required to install kubernetes on compute nodes in order to create a cluster using kubespray
3. Instantiate the binary package installation and get the status of the installation
4. Get application-k8s kubeconfig file
5. BPA is also expected to store any private key and secret information in CSM
6. Install the packages on newly added compute nodes
7. Update package versions in compute nodes that require the update
8. Store private keys 

 For more information on the BPA functions, check out the ICN Akraino project link above

Implementation

We do not intend to make any changes to the existing kubernetes API in order to implement the specifications described in this document. We will simply be extending the Kubernetes API using Custom Resource Definition as described here and then creating a custom controller that will handle the requirements of our provisioning Agent custom resource. 

Overview of Proposed Workflow   

Prerequisites: This workflow assumes that the baremetal CR and  baremetal operator have been created and has successfully installed the compute nodes with Linux OS. It also assumes that the BPA controller is running.

Fig 1: Illustration of the proposed workflow

Workflow Summary/Description

1. Create BPA CRD (Created only once and just creates the BPA resource kind)

2. Create the BPA Custom Resource

3. The BPA Operator continues to watch the k8s API server and once it sees that a new BPA CR object has been created, it queries the k8s API server for the Baremetal hosts lists. The baremetal hosts lists contains information about the compute nodes provisioned including the IP address, CPU, memory..etc of each host.

4. The BPA operator looks into the baremetal hosts list returned and decides on the roles of the compute nodes based on the CPU and memory (Other features may be added as we proceed). The resource requirements for master and worker nodes are defined in the BPA CR object.

5. The BPA operator then creates the hosts.ini file using the assigned roles and their corresponding IP addresses.

6. The BPA operator then installs kubernetes using kubespray on the compute nodes thus creating an active kubernetes cluster. During installation, it would continue to check the status of the installation

7. On successful completion of the k8s cluster installation, the BPA operator would  save the application-k8s kubeconfig file in order to access the k8s cluster and make changes such as software updates or add a worker node for future purposes.

BPA CRD

The BPA CRD tells the Kubernetes API how to expose the provisioning custom resource object. The CRD yaml file is applied using 

“kubectl create -f bpa_v1alpha1_provisioning_crd.yaml”  See below for the CRD definition.

 BPA CRD Yaml File (*_crd.yaml)

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: provisionings.bpa.akraino.org
spec:
  group: bpa.akraino.org
  names:
    kind: Provisioning
    listKind: ProvisioningList
    plural: provisionings
    singular: provisioning
    shortNames:
    - bpa
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 
          type: string
        kind:
          description: 
          type: string
        metadata:
          type: object
        spec:
          type: object
        status:
          type: object
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true

Provisioning Agent Object Definition( *_types,go)

The provisioning_types.go file is the API for the provisioning agent custom resource. 

// ProvisioningSpec defines the desired state of Provisioning
type ProvisioningSpec struct {
        Masters []Master `json:"master,omitempty"`
        Workers []Worker `json:"worker,omitempty"`
        Replicas int32  `json:"replicas,omitempty"`
}

// ProvisioningStatus defines the observed state of 
// Provisioning
type ProvisioningStatus struct {
        // Names of provisioning agent pods when a deployment
        // is running
        PodAgents []string `json"podAgents,omitempty"`
}
// Provisioning is the Schema for the provisionings API
type Provisioning struct {
        metav1.TypeMeta   `json:",inline"`
        metav1.ObjectMeta `json:"metadata,omitempty"`

        Spec   ProvisioningSpec   `json:"spec,omitempty"`
        Status ProvisioningStatus `json:"status,omitempty"`
}

// ProvisioningList contains a list of Provisioning
type ProvisioningList struct {
        metav1.TypeMeta `json:",inline"`
        metav1.ListMeta `json:"metadata,omitempty"`
        Items           []Provisioning `json:"items"`
}

// master struct contains resource requirements for a master 
// node
type Master struct {
        CPU int32      `json:"cpu,omitempty"`
        Memory string  `json:"memory,omitempty"`
}

// worker struct contains resource requirements for a worker node
type Worker struct {
        CPU int32      `json:"cpu,omitempty"`
        Memory string  `json:"memory,omitempty"`
        SRIOV bool     `json:"sriov,omitempty"`
        QAT  bool      `json:"qat,omitempty"`
}

The variables in the ProvisioningSpec struct are used to create the data structures in the yaml spec for the custom resource. Three variables are added to the ProvisioningSpec struct;

1. Masters: This variable will contain an array of Master objects. The master struct as defined in the *-types.go file above contains CPU and memory information, this information would be used by the BPA operator to determine which compute nodes to assign the role of Master to when it gets the baremetal list from the API server.
2. Workers: This variable will contain an array of Worker objects. Similar to the case of the Masters variables, the Worker struct will contain resource requirements for the Worker nodes and the BPA operator will use this information to determine which hosts to assign the role of worker.
3. Replicas: An integer that defines the number of pods that should run when the CR is deployed.

In the ProvisioningStatus struct, the podAgents variable is defined. This variable will display a list of the names of the pods that are part of the provisioning agent deployment when the status of the deployment is queried.

5.3. Sample Provisioning CR YAML file

The YAML file above can be used to create a provisioning custom resource which is an instance of the provisioning CRD describes above. The spec.master field corresponds to the Masters variable in the ProvisioningSpec struct of the *-types.go file, while the  spec.worker field corresponds to the Workers variable in the ProvisioningSpec struct of the *-types.go file and the spec.replica field corresponds to the Replicas variable in the same struct. 

Based on the values above, when the BPA operator gets the baremetal hosts object (Step 5in figure 1), it would assign hosts with 10 CPUs and 4Gi memory the role of master and it would assign hosts with 20CPUs and 8Gi memory the role of worker. 

The replicas spec tells the controller that in this CR deployment, at any instance, there should be 2 pods running the provisioning agent. If a pod dies, the controller reconciles this and immediately starts a new pod running the provisioning agent.

• Open Questions

• How does the BPA operator get the SSH information of the compute hosts ?

• Future Work

      This proposal would make it possible to assign roles to nodes based on the features discovered. Currently, the proposal makes use of CPU, memory, SRIOV and QAT. However the baremetal operator list returns much more information about the nodes, we would be able to extend this feature to allow the operator assign roles based on more complex requirements such as CPU model. This would feed into Hardware Platform Awareness (HPA)

• References

1. https://wiki.akraino.org/pages/viewpage.action?pageId=11995877&show-miniview
2. https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#advanced-topics

I like cheese!