SDEWAN central controller provides central control of SDEWAN overlay networks by automatically configuring the SDEWAN CNFs located in edge location clusters and hub clusters:
- To create secure overlays where each overlay connects application and hub clusters together.
- To allow application connectivity with external entities and entities of other clusters.
System Architecture
SDEWAN central controller includes the following components as showed in below diagram:
- Web UI: a HTML5 based web UI to provide configuration of Application Cluster Registration, Hub Registration, Overlay, Application/Service Registration and Status tracking.
- API Server: Exports Restful API for Application Cluster management, Hub management, Overlay management, Status monitoring management, logging.
- Scheduler Manager: a daemon service which accepts request from API server (through RPC) then generates relevant K8s CRs of SD-EWAN CNFs of various hubs and edges to establish the tunnels.
- SDEWAN Management DB: a database to store information such as edge clusters, hubs, overlays, ip addresses, application/services etc.
System Design
Working Flow
Assumption
IP
- Central Cloud has public IP as CIP
- Traffic Hub has public IP as HIP1 HIP2, ...
- Edge Location may have public IP in one edge node as EIP1, ... or don't have public IP (behind a gateway as EGIP1, ...)\
IPSec Tunnel mode for control plane (e.g. central cloud to k8s API server)
- Central Cloud to Traffic Hub: Host to Host
- Central Cloud to Edge Location:
- Edge location has public IP: Host to Host
- Edge location does not have public IP: Initiator (edge) to Responder (Central cloud)
IPSec Tunnel mode for data plane (for data traffic)
- Edge to Edge: Host to host
- Edge to Hub: Host (edge) to Site (Hub, using edge's subnet as rightsubnet)
- Hub to Hub: Host to Host
Opens:
- Control plane and data plane share the same interface in Hub? Edge Location?
- Control plane IPSec tunnel between Central Cloud with Hub is setup during Hub registeration in Central Cloud
- Control plane IPSec tunnel between Central Cloud with Edge location (with public IP) is setup during edge location registeration in Central Cloud
- Control plane IPSec tunnel between Central Cloud with Edge location (with private IP) is setup during edge location setup (depedency to check: IPsec tunnel for Initiator to Responder requires Responder to be run first)
Environment Setup (Pre-condition)
Central Cloud:
- K8s cluster is setup (by Kud)
- Web UI, API Server, SDEWAN controller, DB service are deployed (through EMCO)
- Central SDEWAN Config Agent and CNF are deployed (through EMCO) with initial configuration (e.g. as Responder for Edge location without public IP, left: CIP, leftsubnet: from IP Address manager?, rightsourceip: from IP Address manager?)
Traffic Hub:
- K8s cluster is setup (by Kud)
- Hub SDEWAN Config Agent and CNF are deployed (through EMCO) with initial configuration (e.g. As Host for Control plane - left: HIP, right: CIP). Note: at this stage, the Central-Hub tunnel is not setup yet.
Edge Location (Public IP):
- K8s cluster is setup (by Kud)
- Edge SDEWAN Config Agent and CNF are deployed (through EMCO) with initial configuration (e.g. As Host for Control plane - left: EIP, right: CIP). Note: at this stage, the Central-Edge tunnel is not setup yet.
Edge Location (Private IP):
- K8s cluster is setup (by Kud)
- Edge SDEWAN Config Agent and CNF are deployed (through EMCO) with initial configuration (e.g. As Initiator for Control plane - left: %any, leftsourceip:%config, right: CIP, rightsubnet:0.0.0.0/0). Note: at this stage, an OIP is assigned to the CNF and the Central-Edge tunnel is set up (to be confirmed)
Opens:
- During current test, IPsec tunnel for Initiator to Responder requires Responder to be run before Initiatior, that means the SDEWAN CNF in Central cloud need to be run as Responder before a edge location (with private IP) setup, and the OIP Address range need to be confgiure first (read from IP address manager?) and can not be updated at run time, does this be expected behavior?
- Need to check how to get the assigned OIP after the tunnel between Central Cloud and Edge Location (with private ip) setup (through strongswan command?), this is required for Ip address manager and cluster register process.
- The registration of edge location information should be done by Admin manually or triggled automatically by EMCO's edge location registration process (assume simaliar information shared)?
Flow: Hub
Register Hub:
- Trigger: Admin add/update hub information in Web UI or Remote Client Call with below informations:
- Name, Description
- Public IP address list
- Managed IP ( ? )
- Shared flag (whether the hub can be shared cross overlays)
- Overlay name
- CertificateId
- Kubeconfig
- Steps:
- Save in DB
- Setup control plane host-host tunnel with Central Cloud (e.g. Add a new IPSec policy in Central Cloud CNF with: left: CIP, right: HIP, CertificateId)
Opens:
- In case multiple public IPs, needs to define which HIP should be used in tunnel with Central Cloud
Flow: Edge Location
Register Edge Location:
- Trigger: Admin add/update edge location information in Web UI or Remote Client Call with below informations:
- Name, Description
- External IP address (empty if no public IP)
- Flag as force Hub connectivity (Valid if external public IP is not empty)
- Flag as use Hub for internet connectivity
- Flag as Dedicated SFC
- Number of overlay IP addresses
- CertificateId
- Kubeconfig
- Steps:
- Save in DB
- if public ip is not empty, Setup host-host tunnel with Central Cloud (e.g. Add a new IPSec policy in Central Cloud CNF with: left: CIP, right: EIP, CertificateId)
- if public ip is empty, no more actions (suppose the tunnel had been setup after edge location setup)
Opens:
- the OIP for control plane (with Central Cloud) will be generated by Centran Cloud responder, shall this OIP be used for data plane (e.g. edge1↔hub↔edge2) or new OIP should be created (e.g. use Hub as responder) in Add-edge-location flow in overlay, and the Number of overlay IP address will be used to block Add-edge-location flow if exceeded?
Flow: Overlay
Add-basic-information:
- Trigger: Admin add/update edge location information in Web UI or Remote Client Call with below informations:
- Name, Description
- CertificateId
- Overlay IP ranges
- Steps:
- Save in DB
Opens:
- Can overlay IP ranges be same for different overlay? (Suppose "yes" as supposing the edges belongs to different overlays will not communicate even share the same hub)
- How to avoid EMCO deploy different microservices of one application into different overlays?
Add-hub:
- Trigger: Admin add/update hub overlay information in Web UI or Remote Client Call with below informations:
- Overlay name
- hub name
- Hub ip (if hub has more than 1 public IPs)
- Hub overlay ip ranges
- Steps:
- Save hub list information in DB
- Setup hub-hub tunnel (data plane): e.g. left: HIP1, right: HIP2, overlay CertificateId
- Setup hub as responder of edge-hub tunnel (data plane): e.g. left: HIP, leftsubnet: Hub overlay ip subnet, rightsourceip: Hub overlay ip ranges
Opens:
- Does it need define overlay ip ranges special for a hub or use overlay's ip range directly?
- Can 2 Hub setup 2 channels with different masks/interface ids (Need check)?
Add-edge-location:
- Trigger: Admin add/update application cluster overlay information in Web UI or Remote Client Call with below informations:
- Overlay name
- edge location name
- connected Hub name(s)
- Steps:
- Save application cluster overlay information in DB
- Setup edge-hub tunnel with first hub (data plane): e.g. as Initiator - left: %any, leftsourceip:%config, right: HIP, rightsubnet:0.0.0.0/0, overlay CertificateId
- Get the assigned OIP, save to DB and broadcast to other hubs (add to exclude list of its responder - Need to check how to do it)
- Setup edge-hub tunnel with remain hub (data plane): e.g. as host-host tunnel, left: OIP, right: HIP, overlay CertificateId
Opens:
- Suppose a edge location can only belong to one overlay at the same time?
- Can edge location connected to more than 1 hubs? if yes, Can it be assigned multiple OIPs from different hubs?
- For edge with public ip, does it need setup Initiator-responder tunnel or host-host tunnel with hub?
Flow: Application Connection
Add-application-rule:
- Trigger: Admin add/update application crule in Web UI or Remote Client (ONAP4K8s connectivity orchestrator?) Call with below informations:
- Application name
- Deployed edge location name
- Firewall/SNAT/DNAT
- Steps:
- Save application rule information in DB
- Setup openwrt rules in SDEWAN CNF of edge location
Add-application2application-rule:
- Trigger: Admin add/update application crule in Web UI or Remote Client (ONAP4K8s connectivity orchestrator?) Call with below informations:
- Application1 name
- Application2 name, port
- Edge location1 names for application1
- Edge location2 names for application2
- Steps:
- Save application2application rule information in DB
- Setup openwrt SNAT rule of SDEWAN CNF for edge location1
- Setup openwrt DNAT rule of SDEWAN CNF for edge location2
- Setup ip route rule in hubs between edge location1 and edge location2
Opens:
- The registration of application/microservice connection information is done by Admin manually or triggled automatically by EMCO's deployment process (assume simaliar information shared)?