Page History

...

In ICN's SDWAN usages, SFC (Service Function Chain) is designed to support Corp networks to connect to external internet with security connection. The SFC includes Security VNF (e.g. firewall etc.), WAN Opt CNF and SDWAN VNF/CNF, and SDWAN module is worked as software defined router which can be used to defined the rules when connect to external internet. Below diagram shows where SDWAN module located in the whole system.

Image Removed

Basic

...

Technologies

OpenWRT

The OpenWRT Project (https://openwrt.org/) is an open source project based on Linux, and it is primarily used on embedded devices to route network traffic. There are more than 3500 software packages which can be installed on OpenWRT via opkg package management system. OpenWRT provides both docker image and VM image to support virtualization solution (https://openwrt.org/docs/guide-user/virtualization/start). In ICN, we run OpenWRT in container.

...

ovn4nfv-k8s-plugin is a CNI plugin based on ovn. It can work together with Multus CNI to add multiple interfaces for the pod. One of the interfaces is the Multus default interface, it could be flannel, calico, etc. The other interfaces are added by ovn4nfv-k8s-plugin according the the pod annotation. With ovn4nfv-k8s-plugin, we can create virtual network in run-time. Also we can connect the pod with the provider network, this is important for CNF.

Design

...

Implementation

SDWAN module is implemented as CNF instead of VNF for better performance and proving of CNF can also be effect way to deploy SDWAN

SDWAN will leverage OpenWRT for it functionality:

Base Image: x86-generic rootfs image (http://downloads.openwrt.org/releases/18.06.4/targets/x86/generic/)
Packages:
- mwan3: multiple wans configuration and management
- luci-app-mwan3: web configuration interface for mwan3
- uhttp: web server for luci services

Deployment

Helm will be used to deploy SDWAN CNF as pod on nodes with external network connection with below parameters:

...

WAN interface definition (e.g. interface, member etc.)

...

config interface 'wan'
	option enabled '1'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option reliability '2'
	option count '1'
	option timeout '2'
	option interval '5'
	option down '3'
	option up '8'

config member 'wan_m1_w3'
	option interface 'wan'
	option metric '1'
	option weight '3'

Overview

SDEWAN is a solution to enable SDWAN functionalities include multiple WAN link support, WAN traffic management, NAT, firewall, IPSec and Traffic shaping etc. with focus to address the challenges when applying on edge computing environment like resource limitation, edge overlays, traffic sanitization, automation and cost sensitive etc. The solution includes below components:

SDEWAN CNF: implemented based on OpenWRT, it enhances OpenWRT Luci web interface with SDEWAN controllers to provide Restful API for network functions' configuration and control.
SDEWAN CRD Controller: implemented as k8s CRD Controller, it manages CRDs (e.g. Firewall related CRDs, Mwan3 related CRDs and IPSec related CRDs etc.) and internally calls SDEWAN Restful API to do CNF configuration.
Overlay Controller: provides central control of SDEWAN overlay networks by automatically configuring the SDEWAN CNFs through SDEWAN CRD controller located in edge location clusters and hub clusters.

Initial Traffic rules (e.g. policy, rule etc.)

...

config policy 'wan_only'
	list use_member 'wan_m1_w3'

config rule 'sticky_even'
	option src_ip '0.0.0.0/0.0.0.1'
	option dest_port '443'
	option proto 'tcp'
	option use_policy 'wan_only'

Runtime Configuration

SDWAN traffic rules and WAN interfaces are required to be configured at runtime through Restful API interface.

OpenWRT luci invoking mechanism:

(1) logon: POST /cgi-bin/luci with luci_username and luci_password to get sysauth Cookie

e.g. wget --post-data "luci_username=root&luci_password=" http://192.168.56.2/cgi-bin/luci/

response header will include sysauch cookie, like: "Set-Cookie: sysauth=e5b2e5c2ae0099c078bb3cb72052ed95;"

(2) Call luci http service with sysauth Cookie

e.g. wget --header="Cookie:sysauth=e5b2e5c2ae0099c078bb3cb72052ed95" http://192.168.56.2/cgi-bin/luci

Mwan3 supported services:

interface_status: GET /cgi-bin/luci/admin/status/mwan/interface_status

...

Sample response:

"interfaces":{"wan":{"running":true,"score":0,"track_ip":[{"status":"down","latency":0,"packetloss":0,"ip":"208.67.220.220"},{"status":"down","latency":0,"packetloss":0,"ip":"208.67.222.222"},{"status":"down","latency":0,"packetloss":0,"ip":"8.8.4.4"},{"status":"down","latency":0,"packetloss":0,"ip":"8.8.8.8"}],"lost":536,"status":"offline","age":5,"turn":134},"wanb6":{"running":false,"score":0,"track_ip":[],"lost":0,"status":"","age":0,"turn":0},"wanb":{"running":false,"score":0,"track_ip":[],"lost":0,"status":"","age":0,"turn":0},"wan6":{"running":false,"score":0,"track_ip":[],"lost":0,"status":"","age":0,"turn":0}},"connected":{"ipv6":["Members:","fe80::\/64"],"ipv4":["Members:","224.0.0.0\/3","192.168.56.2","127.0.0.0\/8","127.255.255.255","192.168.56.0\/24","10.0.3.0\/24","10.0.3.15","127.0.0.0","192.168.56.255","127.0.0.1","192.168.56.0","10.0.3.0","10.0.3.255"]}

...

Note: configuration for WAN's interface, member, policy and rule are supported in the web UI, but with no direct restful API available. It needs implement luci cgi plugin to provide restful API for WAN configuration. In ICN, one interface is implemented as POC to support running commands in CNF through Rest API call.

command:

POST /cgi-bin/luci/admin/config/command

Execute commands in OpenWRT CNF

Normal Response Code: 200

Error Response Code: 400

Request:

...

SDWAN CNF Controller Implementations

The Akraino-SDEWAN-OutOfBoxNetworking.pptx describes the whole SDEWAN solution design.

ICN R3 release will focus on implementing the SDEWAN CNF and EWAN config Agent then designing the End-to-End demo with other components described in the SDEWAN solution.

Below diagrams describe CRD definition and the interaction the calling sequence of SDEWAN Conf Mgr, K8s, Sdewan CNF and Sdewan Conf Agent.

Sdewan CRD definition:

Image Removed

Calling Sequence:

Image Removed

1. System Deployment process

ICN deploys K8s cluster and installs kud addon: (1) Multus CNI plugin as default CNI plugin (2) ovn4nfv CNI plugin (3) Sdewan Conf Agent (sdewan-operator) as deployment
- Note: Sdewan-operator includes (1) Sdewan Controller (monitor Sdewan CR) (2) Mwan3conf Controller (monitor Mwan3Conf CR) (3) FirewallConf Controller (monitor FirewallConf CR) (4) IPSec Controller (Monitor IpSec CR)
Admin (or SDEWAN Conf manager?) creates (1) Network CR (to setup OVN virtual network) (2) Provider Network CR (to setup provider network by configuration network interface on each node)

2. Create SDEWAN CNF (Pod) process

SDEWAN conf mgr creates Mwan3Conf CR (or FirewareConf CR, IpSecConf CR), the CRs (for Mwan3Conf CR, it defined the mwan3 policy/rule) are saved in k8s etcd as K8s resources
SDEWAN conf mgr creates Sdewan CR with below information:
- Node: the CNF pod should be created on which node
- Interfaces: include (1) internal network interface which connect to OVN virtual network (2) provider network interface which connect to provider network
- Configuration: the name of pre-defined Mwan3Conf/FirewallConf/IpSecConf CR
Sdewan Controller (running inside Sdewan Conf Agent) gets the notification of new-created Sdewan CR, call K8s API to (1) create Sdewan CNF (Pod and Service) on required Node (through NodeSelector) (2) Create ConfigMap which stores logical network interface information(used to generate /etc/config/network file in Sdewan CNF container).
- Note: OpenWRT applications (such as mwan3, firewall, ipsec etc.) do not use system network interfaces (e.g. "eth0", "net1" which can be listed by "ip a") directly, instead, it uses the logical interfaces (such as "lan", "wan1" etc.), and the logical interfaces are map to real network interfaces in file /etc/config/network
K8s creates the Sdewan CNF pod and call ovn4k8s CNI plugin to attach required network interfaces (defined in Sdewan CR) with the Pod
When the Pod is ready(through k8s readiness check), Sdewan Controller (running inside Sdewan Conf Agent) call the rest API (through Node's FQDN) to (1) login (2) Set configuration (defined in Mwan3Conf, FirewallConf or IpSecConf) to setup initial rule inside the CNF (3) restart Mwan3 (or Firewall, IpSec) service to apply the rules in the CNF
- Note: the configuration rules can be updated/added/deleted at runtime in Update/Delete Rule process

3. Update/Delete Rule process (use Mwan3conf as example)

SDEWAN conf mgr updates Mwan3Conf CR (or FirewareConf CR, IpSecConf CR), the CR is saved inside K8s etcd as resource
K8s notifies Mwan3Conf controller (run inside Sdewan Conf Agent) the CR update/delete event
Mwan3Conf controller (run inside Sdewan Conf Agent) finds all Sdewan CRs which uses this Mwan3Conf (through Sdewan CR's Mwan3Conf property), then update the status of the found Sdewan CRs to trigger Sdewan controller's Reconcile which updates/deletes new Mwan3Conf and restarts Mwan3 service to apply the change.

4. Delete SDEWAN CNF (Pod) process

SDEWAN conf mgr deletes Sdewan CR
K8s notifies Sdewan controller (run inside Sdewan Conf Agent) the CR delete event
Sdewan controller (run inside Sdewan Conf Agent) finds the Sdewan CNF owned by this deleted CR, call k8s API to delete the CNF

Timeline

...

Start/stop/restart/reload SDWAN service, includes: mwan3, firewall/NAT, IpSec.

Reference: SDEWAN CNF#SDEWANService

...

Support MWAN3 rule/policy configuration.

Reference: SDEWAN CNF#MWAN3

OpenWRT Reference: https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3

...

Design: Feb.26

Implementation: Mar.12

...

WW08: Initial design Done

WW09: Implementation - 50%

WW10: 80%

WW11: done

...

Support firewall configuration for zone (general rule for a group of interfaces), forwarding (iptables forward), rule, redirect (DNAT/SNAT).

Reference: SDEWAN CNF#Firewall

OpenWRT Reference: https://openwrt.org/docs/guide-user/firewall/firewall_configuration

...

Design: Feb.26

Implementation: Mar.18

WW08: Initial design Done

WW09: design done (to be reviewed)

WW10/11/12: 90%

...

Support IPSec configuration for remote site, proposal.

Reference: https://wiki.akraino.org/display/AK/IPSec+Design#IPSecDesign-IPSecRestAPI

OpenWRT Reference: https://openwrt.org/docs/guide-user/services/vpn/ipsec/strongswan/start

(Note: OpenWRT Wiki page is out-of-date compare to 18.06 implementation which we used and the current design is based on openwrt ipsec code directly)

...

leverage kud to setup 3 clusters (Hub, edge1, edge2)

use pre-defined yaml file (with network interface information and rules definition) to create Sdewan CNF

use linux shell script to call CNF Rest API (e.g. update rule, restart service etc.)

shell script to verify ms connectivity in different edge cluster

...

R3.1

POC to verify the flow for n:m label matching between CR instances and CNF instances (e.g. a CR can apply to multiple CNF and a CNF can have multiple CR)

...

Redesigned in R3.1

Define a SDWAN CNF with mwan3, firewall and IPSec configuration

Reference: Sdewan CRD Controller

...

WW21

...

Redesigned in R3.1

Define MWAN3 configuration (policy, rule)

Reference: Sdewan CRD Controller

...

Redesigned in R3.1

Define Firewall CRD (zone, forwarding, rule, redirect (NAT))

...

Redesigned in R3.1

Define IPSec CRD (remote site, proposal)

Reference: https://wiki.akraino.org/display/AK/IPSec+Design#IPSecDesign-IPSecCRD

Scenario design: SD-EWAN Scenarios

...

Redesigned in R3.1

MWAN3 CRD/Restful API integration

...

Redesigned in R3.1

Firewall CRD/Restful API integration

...

Redesigned in R3.1

IPSec CRD/Restful API integration

...

SDEWAN CNF

Sdewan CRD Controller

...

Space shortcuts

Page tree

Versions Compared

Old Version 42

New Version Current

Key

Basic

Technologies

OpenWRT

Design

Implementation

Deployment

Overview

Runtime Configuration

SDWAN CNF Controller Implementations

Timeline

SDEWAN CNF

Sdewan CRD Controller