...
Architecture:
Blocks and Modules
All the green items are existing open source projects. If they require any enhancements, it is best done in the upstream community.
...
- KPM can be borrowed from Multi-Cloud K8S plugin.
- Some part of provisioning controller (mainly location registration) can be borrowed from Multi-Cloud K8S plugin.
- Provisioning controller might use Tekton or Argo for workflow management.
Infra-local-controller:
...
Since, there are few K8S cluster, let us define them:
- infra-global-controller-K8S : This is the K8S cluster where infra-global-controller related containers are run.
- infra-local-controller-K8S: This is the K8S cluster where the infra-local-controller related containers are run.
- application-K8S : These are K8S clusters where application workloads are run.
Infra-local-controller:
"infra-local-controller" is expected to run in bootstrap machine of each location. Bootstrap is the one which installs the required software in compute " is expected to run in bootstrap machine of each location. Bootstrap is the one which installs the required software in compute nodes used for future workloads. Just an example, say a location has 10 servers. 1 server can be used as bootstrap machine and all other 9 servers can be used compute nodes for running workloads. Bootstrap machine is not only installs all required software in the compute nodes, but also is expected to patch and update compute nodes with newer patched versions of the software.
As you see above in the picture, bootstrap machine itself is based on K8S. Note that this K8S is different from the K8S that gets installed in compute nodes. That is, these are two are different K8S clusters. In case of bootstrap machine, it itself is complete K8S cluster with one node that has both master and minion software combined. All the components of infra-local-controller (such as BPA, KPA, Metal3 Metal3 and Ironic) themselves are containers.
...
- As a USB bootable disk: One should be able to get any bare-metal server machine, insert USB and restart the server. What it means is that USB bootable disk shall have basic Linux, K8S and all containers coming up without any user actions. It is also expected to have packages and OS images that are required to provision actual compute nodes. As in above example, these binary, OS and packages are installed on 9 compute nodes.
- As individual entities : As developers, one shall be able to use any machine without inserting USB disk. In this case, developer can choose a machine as bootstrap machine, install Linux OS, Install K8S using Kubeadm and then bring up BPA, KPA, Metal3 Metal3 and Ironic. Then upload packages via RESTAPIs provided by BPA to the system.
- As a KVM/QMEU Virtual machine image: One shall be able to use any VM as bootstrap machine using this image.
Note that Note that infra-local-controller can be run without infra-global-controller. In interim release, we expect that only infra-local-controller is supported. infra-global-controller is targeted for final Akraino R2 release. it is the goal that any operations done in interim release on infra-local-controller manually are automated by infra-global-controller. And hence the interface provided by infra-local-controller is flexible to support both manual actions as well as automated actions.
...
Step 1 and 2 are expected to be taken care using Metal3 and Ironic. Step 3 is expected to be taken care by BPA and Step 4 is expected to be taken care by KPA.Procedure the user would following is this:talking to application-K8S
User experience for infrastructure administrators:
When using URB bootable disk
- Select a machine in the location for bootstrapping.
- Boot up a bootstrap machine using USB bootable disk.
- Via Kubectl to infra-local-controller via Metal3 CRs, make ironic ready for compute nodes to do PXEBOOT and install Linux.
...
- Upload site specific information via BPA CR - Compute nodes, their roles etc...
- Once Linux get installed, Via Kuberctl to BPA (via CR), make BPA install the binary packages (such as Kubelet, docker, kubectl, kubenetes API server for application-K8S)
- Via Kuberctl to BPA, get hold of kubeconfig of application-K8S
- Using this kubeconfig, via kubectl to application-K8S, install the packages that can be done via kubectl (such as Multus, OVN Controllers, Virtlet etc...)
As a developer:
- Select a machine in the location for bootstrapping.
- Install Linux OS
- Install Kubernetes on this machine using Kubeadm or any of your favorite tool
- Upload all binary packages, Linux OSes to be installed in compute nodes using for applications.
- Upload site specific information - Compute nodes, their roles etc...
- Once Linux get installed, Via Kuberctl to BPA (via CR), make BPA install the binary packages (such as Kubelet, docker, kubectl, kubenetes API server for application-K8S)
- Via Kuberctl to BPA, get hold of kubeconfig of application-K8S
- Using this kubeconfig, via kubectl to application-K8S, install the packages that can be done via kubectl (such as Multus, OVN Controllers, Virtlet etc...)
- Make a USB bootable disk for administrators to use in real deployments.
- Make a VM image for administrators to use in real deployments.
Binary Provisioning Agent (BPA)
BPA job is to install all packages that can't be installed using kubectl to application-K8S. Hence, BPA is normally used right after compute nodes get installed with Linux operating system, before installing kubernetes based packages. BPA is also an implementation of CRD controller of infra-local-controller-k8s. We expect to have following CRs:
- To upload site specific information - compute nodes and their roles
- To instantiate the binary package installation.
- To get hold of application-K8S kubeconfig file.
- Get status of installation
BPA also provides some RESTful API for doing following:
- To upload binary images that are used to install the stuff in compute nodes.
- To upload Linux Operating system that are needed in compute nodes.
- Get status of installation of all packages as prescribed before.
Since compute nodes may not have Internet connectivity
- Ensure that BPA also acts as local Docker Hub repository and BPA needs to ensure that all K8S container packages (that need to be installed on the application-K8S) are served locally here.
- BPA also need to ensure that it configures the docker to get hold of packages from this local repository.
BPA also takes care of: (After interim release)
- When new compute node is added, once administrator adds new compute node in the site list, it shall take care of installing the packages.
- If a new binary package version is uploaded, it shall take care of figuring out the compute nodes that require this new version and update that compute node with the new version.
Implementation suggestions:
KuD needs to be broken into piecies
- KuD that installs basic packages via Kubespray and packages that are not containerized. BPA can inherit this code.
- KuD that acts as private docker hub repository. BPA can inherit this code.
- KuD that builds the packages from the source code - this needs to be done outside of BPA and binary packages and container packages that result from these are expected to be part of USB bootable disk.
- KuD that brings containerized packages : This needs to be taken care as a script on top of infra-local-controller.
Solution Details
...
Global ZTP:
Global ZTP system is used for Infrastructure provisioning and configuration in ICN family. It is subdivided into 3 deployments Cluster-API, KuD and ONAP on K8s.
...