...
- To create secure overlays where each overlay connects application and hub clusters together.
- To allow application connectivity with external entities and entities of other clusters.
System Architecture
SDEWAN central controller The system includes the following components micro-services as showed in below diagram:
- SDEWAN Central Controller:
- API Router:
...
- provides REST API router for SDEWAN Central Controller
- OverlayObjectManager: overlay registration, generate overlay root cert
- HubObjectManager: hub registration and setup hub connection mesh
- DeviceObjectManager: device/cluster registration and setup device connection mesh (if device has public IP)
- HubDeviceObjectManager: setup connection between hub and device
- IPRangeObjectManager: ip range registration and allocate/free overlay ip for device
- ProposalObjectManager: proposal registration
- DeviceConnManager: only support GET, query connection status of device
- HubConnObjectManager: only support GET, query connection status of hub
- Observability framework: system status monitoring, including connection status, CNF status etc.
- Rsync
- Web UI: a HTML5 based web UI to provide configuration of Application Cluster Registration, Hub Registration, Overlay, Application/Service Registration and Status tracking.
- API Server: Exports Restful API for Application Cluster management, Hub management, Overlay management, Status monitoring management, logging.
- Scheduler Manager: a daemon service which accepts request from API server from SDEWAN Central Controller (through RPC) then generates deploy relevant K8s CRs of SD-EWAN CNFs of various hubs and edges to establish the tunnels.
- SDEWAN Management Mongo DB: a database to store information such as edge clusters, hubs, overlays, ip addresses, application/services etc.
- Etcd: a metadata database to exchange configuration information between SDEWAN Central Controller and Rsync
System Design
Working Flow
Assumption
IP
- Central Cloud has public IP as CIP
- Traffic Hub has public IP as HIP1 HIP2, ...
- Edge Location (Device) may have public IP in one edge node as EIP1, ... or don't have public IP (behind a gateway as EGIP1, ...)\
IPSec Tunnel mode Connection for control plane (e.g. central cloud to k8s API server):
- Central Cloud to Traffic Hub: Host to HostDirect connection through Hub's public IP
- Central Cloud to Edge Location:
- Edge location has public IP: Host to Host Direct connection through Edge Location's public IP
- Edge location does not have public IP: Initiator (edge) to Responder (Central cloud)Using Edge Location owned hub's SDEWAN CNF as proxy
IPSec Tunnel mode for data plane (for data traffic)
- Edge to Edge: Host to host
- Edge to Hub: Host (edge) to Site (Hub, using edge's subnet as rightsubnet)
- Hub to Hub: Host to Host
Open:
...
...
Environment Setup (Pre-condition)
...
- K8s cluster is setup (by Kud)
- Web UI (Optional), API Server, SDEWAN controllerRsync backend, DB service are deployed (manually or through EMCO)
- Central SDEWAN Config Agent and CNF are deployed (through EMCO) with initial configuration (e.g. as Responder for Edge location without public IP, left: CIP, leftsubnet: from IP Address manager?, rightsourceip: from IP Address manager?)
Traffic Hub:
- K8s cluster is setup (by Kud)
- Hub SDEWAN Config Agent CRD Controller and CNF are deployed (through EMCO) with initial configuration (e.g. As Host for Control plane - left: HIP, right: CIP). Note: at this stage, the tunnel is not setup yetNAT: enable DNAT for k8s API service and Istio Ingress service).
Edge Location (With Public IP):
- K8s cluster is setup (by Kud)
- Edge SDEWAN Config Agent SDEWAN CRD Controller and CNF are deployed (through EMCO) with initial configuration (e.g. As Host for Control plane - left: EIP, right: CIP). Note: at this stage, the tunnel is not setup yetNAT: enable DNAT for k8s API service and Istio Ingress service).
Edge Location (With Private IP):
- K8s cluster is setup (by Kud)
- Edge SDEWAN Config Agent SDEWAN CRD Controller and CNF are deployed (through EMCO) with initial configuration (e.g. As NAT: enable DNAT for k8s API service and Istio Ingress service; IPSec: as Initiator for Control plane - left: %any, leftsourceip:%config, right: CIPOwned Hub's HIP, rightsubnet:0.0.0.0/0). Note: at this stage, an OIP is assigned to the CNF and the tunnel is set up
Open:
- During current test, IPsec tunnel for Initiator to Responder requires Responder to be run before Initiatior, that means the SDEWAN CNF in Central cloud need to be run as Responder before a edge location (with private IP) setup, and the OIP Address range need to be confgiure first (read from IP address manager?) and can not be updated at run time, does this be expected behavior?
- Need to check how to get the assigned OIP after the tunnel between Central Cloud and Edge Location (with private ip) setup (through strongswan command?), this is required for Ip address manager and cluster register process.
- The registration of edge location information is done by Admin manually or triggled automatically by EMCO's edge location registration process (assume simaliar information shared)?
Flow: Register Hub
Flow: Register Edge Location
Flow: Register Application Service
Open:
- The registration of application/microservice information is done by Admin manually or triggled automatically by EMCO's deployment process (assume simaliar information shared)?
Flow: Register Overlay
Error handling
DB Schema
Restful API definition and Back-End flow
Resource | Description | URL | Fields | Back-End flow |
---|---|---|---|---|
Overlay | Define a group of edge location clusters (devices) and hubs, a overlay is usually owned by one customer and full mesh connections are setup automacally between hub-hub and device-device (with public IPs) | /scc/v1/overlays |
| Registration:
|
Proposal | Define proposals which can be used for IPsec tunnel in this overlay | /scc/v1/overlays/{overlay-name}/proposals |
| Registration:
|
Hub | Define a traffic Hub in an overlay | /scc/v1/overlays/{overlay-name)/hubs |
| Registration:
|
IPRange | Define the overlay IPrange which will be used as OIP of devices | /scc/v1/overlays/{overlay-name}/ipranges |
| Registration:
|
Device | Define a edge location device information which may be a CNF, VNF or PNF | /scc/v1/overlays/{overlay-name}/devices |
| Registration:
|
Hub-device connection | Define a connection between hub and device | /scc/v1/overlays/{overlay-name}/hubs/{hub-name}/devices/{device-name} |
| Create:
Todo: Confirm "ip route" rule for OIP in this hub and all other hub are setup automatically or need new CR to execute linux shell in host |
Error handling
DB Schema
Module Design
Task Breakdowns
Tasks | Due | Owner | Status | Description |
Scheduler Manager | ||||
-- Overlay: Setup tunnels for hubs and edges | Generates relevant K8s CRs of SD-EWAN CNFs of various hubs and edges to establish the tunnels | |||
-- IP Address manager | Assigns/frees IP addresses from "overlay IP ranges" and dedicates them to that cluster | |||
-- Application connectivity scheduler | Creates K8s resources required to be pushed into the edges and corresponding traffic hubs to facilitate the connectivity | |||
-- Resource Synchronizer | ||||
-- CNF | ||||
API Server | ||||
-- Rest API Backend | Rest API server framework | |||
-- DB Backend | Proxy to DB | |||
-- Application Cluster management | ||||
-- Hub management | ||||
-- Overlay management | ||||
-- Status monitoring management | ||||
-- logging | ||||
Web UI | ||||
-- Web UI framework | ||||
-- Application Cluster Registration | ||||
-- Hub Registration | ||||
-- Overlay | ||||
-- Application/Service Registration | ||||
-- Status tracking | ||||
EMCO plugin for SDEWAN | ||||
E2E Integration | Integration test of overall system |
...