Overview
ICN BP family intends to address deployment of workloads in a large number of edges and also in public clouds using K8S as resource orchestrator in each site and ONAP-K8S as service level orchestrator (across sites). ICN also intends to integrate infrastructure orchestration which is needed to bring up a site using bare-metal servers. Infrastructure orchestration, which is the focus of this page, needs to ensure that the infrastructure software required on edge servers is installed on per-site basis, but controlled from a central dashboard. Infrastructure orchestration is expected to do the following:
- Installation : First-time installation of all infrastructure software.
- Keep monitoring for new servers and install the software based on the role of the server machine.
- Patching: Continue to install the patches (mainly security related) if new patch release is made in any one of the infrastructure software packages.
- May need to work with resource and service orchestrators to ensure that workload functionality does not get impacted.
- Software updates: Updating software due to new releases.
infra-global-controller: If infrastructure provisioning needs to controlled from a central location, this component is expected to be brought up in one location. This controller communicates with infra-local-controller (which is kind of agent) that perform the actual software installation/update/patch and provisions the software or BIOS etc...
infra-local-controller: Typically sits in each site in a bootstrap machine. Typically provided as bootable USB disk. It works in conjunction with the infra-global-controller. Note that, if there is no requirement to manage the software provisioning from a central location, then infra-global-controller is brought up along with the infra-local-controller.
User experience needs to be as simple as possible and even novice user shall be able to set up a site
- 1st-time installation:
- User procures set of machines or racks.
- User connects them together with switches.
- User uses the USB or other mechanisms to boot a machine (call it as bootstrap machine) with infra-local-controller stack.
- User provides infra-global-controller FQDN/IP address for local controller to reach to it (Note that if global controller is in the same machine as the local controller, loopback IP address can be used to communicate - this is default option).
- User updates the inventory of machines, defines the role of each machine and inform the central ZTP system (infra-global-controller) or infra-local-controller (bootstrap) stack using its provided UI or via API.
- System installs the software and verifies the installation of each software component by running tests that are part of ICN-infra.
- User/Entity gets informed when all the machines are installed with the software.
- Addition of new server
- User updates the site inventory with its role to the infra global controller using its UI.
- System expected to install the software.
- System expected to verify the installation of software using tests that are part of bootstrap stack.
- User/entity gets informed when the machine is successfully brought online.
- Deletion of existing server:
- User informs the infra-global-controller to bring down the server.
- System removes the software and cleans up any local disk and other persistent systems
- User/Entity gets informed that server can be taken off from the network and disposed of.
- Patching
- User or external patch system informs the infra-global-controller that new patch(es) is available for a given software package (or packages).
- User or external patch system also informs whether the patch or patches require a restart of the process or kernel.
- System then takes care of patching every server that has these software packages.
- If the software package impacts the workloads:
- Informs the local workload (resource orchestrator) to not deploy new workloads or move existing workloads. Many resource level orchestrator provides a way to decommission the server on a temporary basis.
- Ensures that there are no workloads on the system.
- Installs the patch and do needed restarts.
- Informs the local orchestrator to put the server back in the pool.
- Updating
- User or external update system informs the infra-global-controller that a new software version is available for a current software package
- User or external update system should inform the update is minor or major. A major update required to remove completely the old version, and then install the new version
- Infra-global-controller based the update nature then takes care of update as follows
- Reschedule the existing workload in the server that required an update to the other server in the cluster
- Remove the server from the local orchestrator cluster, provide the update software and reconnects it back
Akraino's "Integrated Cloud Native NFV & App Stack" (ICN) Blueprint is a Cloud Native Compute and Network Framework(CN-CNF) to integrated NFV's application to the de-facto standard and setting a framework to address 5G, IOT and various Linux Foundation edge use case in Cloud Native.
ICN has ONAP as the Service Orchestration Engine(SOE) and the Cloud Native(CN) projects such as Kubernetes for Resource Orchestration Engine(ROE), Prometheus as the monitoring and alerting, OVN as the SDN controller, Container Network Interface(CNI) for Orchestration Networking, provides networking between the clusters, Envoy for Service proxy, Helm and Operators for package management and Rook for storage. The framework stack specifics the best configuration methodology, enables development projects, installation scripts, software package to bind CNCF and LF edge use cases together.
This document break downs the hardware requirements, software ingredient, Testing and benchmarking for the R2 and R3 release for and provides overall picture toward blue print effect in Edge use cases.
Goals
- Generic: Infrastructure Orchestration shall be as generic. Even though this work is being done on behalf of one BP (MICN), infrastructure orchestration shall be common across all BPs in the ICN family. Also, it shall be possible to use this component in other BPs outside of ICN family.
- Leverage open source projects:
- Leverage cluster-API for infra-global-controller. Identify gaps and provide fixed and also provide UI/CLI for good user experience.
- Leverage Ironic and metal3 for infra-local-controller to do bare-metal provisioning. Identify any gaps to make it work with Cluster-API.
- Leverage KuD in infra-local-controller to do Kubernetes installation. Identify any gaps and fix them.
- Figure out ways to use the bootstrap machine also as workload machine (Not in scope for Akraino-R2)
- Flexible and Extensible :
- Adding any new package in future shall be a simple addition.
- Interaction with workload orchestrator shall not be limited to K8S. Shall be able to talk to any workload orchestrator.
- Data Model driven:
- Follow CRD models as much as possible.
- Security:
- Infra-global and infra-local controller may have privileged access to secrets, keys etc.. Shall ensure to protect them by putting them in HW RoT or at least ensure that they are not visible in clear in HDD/SSDs.
- Redundancy: Infra-global controller shall be redundant, especially, if it used to manage multiple sites.
- Performance:
- Shall be able to complete the first time installation or patching across multiple servers in a site shall be in minutes < 10minutes for 10 server site. (May need to ensure that jobs are done in parallel - Multi-threading of infra-local-controller).
- Shall be able to complete the patching across sites shall be done in <10 minutes for 100 sites.
Architecture:
Simplified ICN layout
Blocks and Modules
Software components
Gaps
Solution
Overview
Flows & Sequence Diagrams
infra-global-controller bring up
New site bring up
New server bring-up in existing site
Patch on a software package
Patch on multiple software packages
Server decommissioning
Roadmap
Initial implementation (End of July, 2019)
Akraino R2 release
Akraino R3 release
Future releases