This document describes the use of Public Cloud Edge Interface (PCEI) implemented based on Edge Multi-Cluster Orchestrator (EMCO) for deployment of Public Cloud Edge (PCE) Apps from multiple clouds (Azure and AWS), deployment of a 3rd-Party Edge (3PE) App (an implementation of ETSI MEC Location API App), as well as the end-to-end operation of the deployed PCE Apps using simulated Low Power Wide Area (LPWA) IoT client.
The end-to-end validation environment and flow of steps is shown below:
Description of components of the end-to-end validation environmen:
Note that P1, P3, P5/P7 Reference Points are shown to illustrate alignment with general PCEI architecture.
The end-to-end PCEI validation steps are described below:
For performing Steps 1 and 2 please refer to the PCEI R4 Installation Guide.
Please refer to the PCEI R4 Installation Guide to deploy EMCO and Edge K8S Clusters.
Due to current limitations in EMCO functionality, it is recommended that VNC is used to access the EMCO UI via a Web Browser as shown in the diagram below:
In order to be able to access the EMCO cluster and the Edge K8S cluster via ssh directly from a laptop as shown in the diagram above, please copy the ssh key used in the PCEI Installation to you local machine:
### copy id_rsa key from deployment host to be able to ssh to eco and cluster vm’s sftp onaplab@10.121.1.12 get ~/.ssh/id_rsa pcei-emco ### ssh to VMs ssh -i pcei-emco onaplab@10.121.7.145 |
In order to access EMCO UI please install VNC Viewer on your local laptop and configure it to connect to the VNC server that was installed on the EMCO Host Server (refer to the PCEI R4 Installation Guide for VNC server installation instructions).
Connect to the VNC server, start the Browser and point it to the EMCO VM (amcop-vm-01) IP address and port 30480:
http://10.121.7.145:30480 |
You should be able to connect to EMCO UI as shown below:
The following Network Controllers must be provisioned in EMCO:
### Add controllers rsync: host : rsync port: 9041 type: < leave blank, not required.> priority: < leave blank, not required> ovnaction: host: ovnaction port: 9053 type: action priority: 1 |
To provision Network Controllers, please use the parameters shown above and EMCO UI "Controllers → Register Controllers" tab:
Fill in the parameters for the "rsync" and the "ovnaction" controllers:
To register the two Edge K8S Clusters created during PCEI installation (refer to PCEI R4 Installation Guide), the cluster configuration files need to be copied from the corresponding VMs to the machine that is used to run the VNC server to access EMCO UI. Please make sure that the ssh key used to access the VMs is present on the machine that runs the VNC server. The example below assumes that the VNC server is running on the Host Server used to deploy EMCO and Edge Cluster VMs
# Determine VM IP addresses: [onaplab@os12 ~]$ sudo virsh list --all Id Name State ---------------------------------------------------- 6 amcop-vm-01 running 9 edge_k8s-1 running 10 edge_k8s-2 running [onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-1 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet1 52:54:00:19:96:72 ipv4 10.121.7.152/27 [onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-2 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet2 52:54:00:c0:47:8b ipv4 10.121.7.146/27 sftp -i pcei-emco onaplab@10.121.7.146 cd .kube get config kube-config-edge-k8s-2 sftp -i pcei-emco onaplab@10.121.7.152 cd .kube get config kube-config-edge-k8s-1 |
Using EMCO UI "Cluster → Register Cluster Provider" tabs, provision Cluster Providers for EDGE-K8S-1 and EDGE-K8S-2 clusters:
Click on the Cluster Provider and click on "Add Cluster". Fill in the Cluster Name and add the config file downloaded earlier. Repeat for the two clusters EDGE-K8S-1 and EDGE-K8S-2:
You should see the following result:
At this point the edge clusters have been registered with the orchestrator and are redy for placing PCE and 3PE apps.
The deployment of Azure IoT Edge cloud native application as a PCE involves the following steps:
Login to your Azure Portal and add Subscription, IoT Hub and Container Registry Resources:
Provision IoT Hub:
Provision IoT Edge under IoT Hub:
Copy the "Primary Connection String" from the IoT Edge parameters. This string will be used later in the values.yaml file for the Azure IoT Edge Helm Charts.
Use this link for information on developing custom software modules for Azure IoT Edge:
The example below is optional. It shows how to build a custom module for Azure IoT Edge to read and decode Low Power IoT messages from a simulated LPWA IoT device. Follow the above link to:
The steps below show how to build custom IoT module for Azure IoT Edge using "LoRaEdgeSolution" code from PCEI repo:
Download PCEI repo to the machine that has VSC and Docker installed (per above instructions):
git clone "https://gerrit.akraino.org/r/pcei" cd pcei ls -l total 0 drwxr-xr-x 8 oberzin staff 256 Dec 24 15:44 LoRaEdgeSolution drwxr-xr-x 3 oberzin staff 96 Dec 24 15:44 iotclient drwxr-xr-x 5 oberzin staff 160 Dec 24 15:44 locationAPI |
Using VSC open the LoRaEdgeSolution folder that was downloaded from PCEI repo.
Add required credentials for Azure Container Registry (ACR) using .env file.
Build and push the solution to ACR as shown below. Righ-click on "deployment.template.json":
The docker image for the custom module should now be visible in Azure Cloud ACR:
Use this link to understand the architecture of Azure IoT Edge cloud native application:
https://microsoft.github.io/iotedge-k8s-doc/architecture.html
For the purposes of this document we use the Host Server on which EMCO has been deployed and on which we run the VNC server to package Helm charts for Azure IoT Edge.
Clone Azure IoT Edge from github:
git clone https://github.com/Azure/iotedge cd iotedge/kubernetes/charts/ mkdir azureiotedge1 cp -a edge-kubernetes/. azureiotedge1/ cd azureiotedge1 ls -al total 24 drwxrwxr-x. 3 onaplab onaplab 79 Dec 24 13:14 . drwxrwxr-x. 5 onaplab onaplab 77 Dec 24 13:14 .. -rw-rw-r--. 1 onaplab onaplab 137 Dec 24 13:02 Chart.yaml -rw-rw-r--. 1 onaplab onaplab 333 Dec 24 13:02 .helmignore drwxrwxr-x. 2 onaplab onaplab 220 Dec 24 13:02 templates -rw-rw-r--. 1 onaplab onaplab 14226 Dec 24 13:02 values.yaml |
Modify values.yaml file to specify the "Primary Connection String" from Azure Cloud generated during IoT Hub/IoT Edge provisioning:
vi values.yaml # Change the line below and save the file provisioning: source: "manual" deviceConnectionString: "PASTE PRIMARY CONNECTION STRING FROM AZURE IOT HUB / IOT EDGE SCREEN" #dynamicReprovisioning: false |
Create a tar file with Azure IoT Edge Helm Charts:
# Make sure to change to the "charts" directory cd .. pwd /home/onaplab/iotedge/kubernetes/charts # zip the "azureiotedge1" directory. Be sure to use "azureiotedge1.zip" file name. tar -czvf azureiotedge1.tgz azureiotedge1/ azureiotedge1/ azureiotedge1/.helmignore azureiotedge1/Chart.yaml azureiotedge1/templates/ azureiotedge1/templates/NOTES.txt azureiotedge1/templates/_helpers.tpl azureiotedge1/templates/edge-rbac.yaml azureiotedge1/templates/iotedged-config-secret.yaml azureiotedge1/templates/iotedged-deployment.yaml azureiotedge1/templates/iotedged-proxy-config.yaml azureiotedge1/templates/iotedged-pvc.yaml azureiotedge1/templates/iotedged-service.yaml azureiotedge1/values.yaml ls -al total 12 drwxrwxr-x. 5 onaplab onaplab 102 Dec 24 13:23 . drwxrwxr-x. 4 onaplab onaplab 31 Dec 24 13:02 .. drwxrwxr-x. 3 onaplab onaplab 79 Dec 24 13:14 azureiotedge1 -rw-rw-r--. 1 onaplab onaplab 8790 Dec 24 13:23 azureiotedge1.tgz drwxrwxr-x. 3 onaplab onaplab 79 Dec 24 13:14 edge-kubernetes drwxrwxr-x. 3 onaplab onaplab 60 Dec 24 13:02 edge-kubernetes-crd |
Copy the infrastructure profile tar file to your home directory on the Host Server. This file is needed to define the Service and the App in EMCO:
cd cp /home/onaplab/amcop_deploy/aarna-stream/cnf/vfw_helm/profile.tar.gz . la -l ls -l total 220088 drwxrwxr-x. 9 onaplab onaplab 138 Nov 24 06:53 aarna-stream drwxrwxr-x. 3 onaplab onaplab 56 Nov 24 06:52 amcop_deploy -rw-r--r--. 1 onaplab onaplab 225356880 Nov 23 14:58 amcop_install_v1.0.zip drwxrwxr-x. 22 onaplab onaplab 4096 Dec 24 13:02 iotedge -rw-rw-r--. 1 onaplab onaplab 263 Nov 23 17:13 netdefault.xml -rw-rw-r--. 1 onaplab onaplab 1098 Dec 26 12:19 profile.tar.gz |
Due to limitations in the current EMCO implementation the following step must be performed manually:
SSH to EDGE-K8S-1 VM:
# Determine VMs IP [onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-1 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet1 52:54:00:19:96:72 ipv4 10.121.7.152/27 # ssh from your laptop ssh -i pcei-emco onaplab@10.121.7.152 |
Deploy Azure CRD:
helm install edge-crd --repo https://edgek8s.blob.core.windows.net/staging edge-kubernetes-crd NAME: edge-crd LAST DEPLOYED: Thu Dec 24 21:43:12 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None kubectl get crd NAME CREATED AT edgedeployments.microsoft.azure.devices.edge 2020-12-24T21:43:13Z |
To define Azure IoT Edge Service in PCEI, connect to EMCO UI and select "Projects → Add Project":
Select the project you just created (PCEI-AZURE-IOT-1 in the above example), select "Add Service", specify the Name and click on "Add App":
Click on Add App tp specify the App Name.
IMPORTANT NOTE: Please ensure that the "App Name" string matches the name of the tar file (without the .tar extension" created earlier with the Helm charts for Azure IoT Edge ("azureiotedge1" in this example".
Click "Create".
You should see the screen below:
Click on the "azureiotedge1" Service and attach application definition files. Add App file "azureiotedge.tgz" and profile file "profile.tar.gz". DO NOT CLICK "SUBMIT".
Select the "Placement" tab on the left.
Select the "EDGE-K8S-1" cluster that was registered earlier:
Click "SUBMIT" in the upper right corner:
After clicking "SUBMIT" you should see the screen below:
To deploy Azure IoT Edge App onto the Edge K8S Cluster (EDGE-K8S-1 in this example), select "Deployment Intent Groups" tab on the left side of the previous screen:
Click on the Blue Down Arrow on the right to deploy Azure IoT Edge App:
After successful deployment, you should see a message in GREEN stating that the deployment has been successful.
SSH to the EDGE-K8S-1 VM and verify the Azure IoT Edge pods started:
ssh onaplab@10.121.7.152 # Verify Azure IoT Edge pods are Running in the default namespace kubectl get pods NAME READY STATUS RESTARTS AGE edgeagent-59cf45d8b9-tc5g9 2/2 Running 1 2m8s edgehub-97dc4fdc8-t5qhf 2/2 Running 0 110s iotedged-6d9dcf4757-h474r 1/1 Running 0 2m17s loraread-d4d79b867-2ft2v 2/2 Running 0 110s # Verify services. Note the TCP port for the loraread-xxxxx-yyyy pod (31230 in the example below): kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE awsggc-service NodePort 10.244.1.182 <none> 8883:30883/TCP 9d edgehub LoadBalancer 10.244.12.125 <pending> 5671:30252/TCP,8883:31342/TCP,443:30902/TCP 3m iotedged LoadBalancer 10.244.36.143 <pending> 35000:32275/TCP,35001:32560/TCP 32d kubernetes ClusterIP 10.244.0.1 <none> 443/TCP 32d loraread LoadBalancer 10.244.3.167 <pending> 50005:31230/TCP |
To verify Azure IoT Edge end-to-end operation, perform the following tasks:
On the Host Server list the IP address for the edge_k8s-1 VM:
sudo virsh domifaddr edge_k8s-1 Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet1 52:54:00:19:96:72 ipv4 10.121.7.152/27 ssh onaplab@10.121.7.152 kubectl get svc loraread NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE loraread LoadBalancer 10.244.3.167 <pending> 50005:31230/TCP 8m28s onaplab@localhost:~$ onaplab@localhost:~$ logout Connection to 10.121.7.152 closed. |
#Clone PCEI repo to a local directory: git clone "https://gerrit.akraino.org/r/pcei" cd pcei/iotclient/ ls -l total 8 -rw-r--r-- 1 oberzin 108493823 3230 Dec 26 16:15 STM32SimAzureEMCO.py # Run the LPWA IoT Client. Specify the IP address of the edge_k8s-1 VM and the port number for the loraread service: python2 STM32SimAzureEMCO.py ENTER SERVER IPv4: "10.121.7.152" ENTER SERVER PORT: 31230 31230 SENDING... 2020-12-26T16:20:47-99 d2 62 21 ######## COUNT: 1 {u'txtime': u'2020-12-26T16:20:47-99', u'datarate': 3, u'ack': u'false', u'seqno': 60782, u'pdu': u'007321E7016700d2026862', u'devClass': u'A', u'snr': 10.75, u'devEui': u'0004A30B001BAAAA', u'rssi': -39, u'gwEui': u'00250C00010003A9', u'joinId': 90, u'freq': 903.5, u'port': 3, u'channel': 6} CLOSED SEND RESULT: None NEXT INTERVAL 2020-12-26T16:20:58-25 dc 62 21 ######## COUNT: 2 {u'txtime': u'2020-12-26T16:20:58-25', u'datarate': 3, u'ack': u'false', u'seqno': 60782, u'pdu': u'007321E7016700dc026862', u'devClass': u'A', u'snr': 10.75, u'devEui': u'0004A30B001BAAAA', u'rssi': -39, u'gwEui': u'00250C00010003A9', u'joinId': 90, u'freq': 903.5, u'port': 3, u'channel': 6} CLOSED SEND RESULT: None |
Note that the LPWA IoT Client is generating Temperature, Humidity and Pressure readings in the encoded format: u'pdu': u'007321E7016700d2026862'.
On the EGGE-K8S-1 cluster (edge_k8s-1 VM) veryfy that the Azure IoT Edge LoRaRead pod is receiving LPWA IoT messages and decoding their contents:
ssh onaplab@10.121.7.152 kubectl get pods NAME READY STATUS RESTARTS AGE edgeagent-59cf45d8b9-tc5g9 2/2 Running 1 26m edgehub-97dc4fdc8-t5qhf 2/2 Running 0 26m iotedged-6d9dcf4757-h474r 1/1 Running 0 26m loraread-d4d79b867-2ft2v 2/2 Running 0 26m kubectl logs loraread-d4d79b867-2ft2v loraread Listening ('Connection address:', ('10.244.64.1', 7117)) ('received data:', '{\n "ack": "false", \n "channel": 6, \n "datarate": 3, \n "devClass": "A", \n "devEui": "0004A30B001BAAAA", \n "freq": 903.5, \n "gwEui": "00250C00010003A9", \n "joinId": 90, \n "pdu": "007321E7016700d2026862", \n "port": 3, \n "rssi": -39, \n "seqno": 60782, \n "snr": 10.75, \n "txtime": "2020-12-26T16:20:47-99"\n}') {u'txtime': u'2020-12-26T16:20:47-99', u'datarate': 3, u'ack': u'false', u'seqno': 60782, u'pdu': u'007321E7016700d2026862', u'devClass': u'A', u'snr': 10.75, u'devEui': u'0004A30B001BAAAA', u'rssi': -39, u'gwEui': u'00250C00010003A9', u'joinId': 90, u'freq': 903.5, u'port': 3, u'channel': 6} 007321E7016700d2026862 00d2 62 21E7 {"Pressure": 867, "TIMESTAMP": "2020-12-26T16:20:47-99", "Humidity": 49, "Temperature": 69.80000000000001, "Device EUI": "0004A30B001BAAAA"} sent! Listening Confirmation[0] received for message with result = OK Properties: {} Total calls confirmed: 1 ('Connection address:', ('10.244.64.1', 24020)) ('received data:', '{\n "ack": "false", \n "channel": 6, \n "datarate": 3, \n "devClass": "A", \n "devEui": "0004A30B001BAAAA", \n "freq": 903.5, \n "gwEui": "00250C00010003A9", \n "joinId": 90, \n "pdu": "007321E7016700dc026862", \n "port": 3, \n "rssi": -39, \n "seqno": 60782, \n "snr": 10.75, \n "txtime": "2020-12-26T16:20:58-25"\n}') {u'txtime': u'2020-12-26T16:20:58-25', u'datarate': 3, u'ack': u'false', u'seqno': 60782, u'pdu': u'007321E7016700dc026862', u'devClass': u'A', u'snr': 10.75, u'devEui': u'0004A30B001BAAAA', u'rssi': -39, u'gwEui': u'00250C00010003A9', u'joinId': 90, u'freq': 903.5, u'port': 3, u'channel': 6} 007321E7016700dc026862 00dc 62 21E7 {"Pressure": 867, "TIMESTAMP": "2020-12-26T16:20:58-25", "Humidity": 49, "Temperature": 71.6, "Device EUI": "0004A30B001BAAAA"} sent! Listening Confirmation[0] received for message with result = OK Properties: {} Total calls confirmed: 2 |
Note that the Azure IoT Edge PCE App is decoding the IoT Client readings from the Low Power encoding into clear text JSON format and forwarding the decoded readings to the Azure IoT Hub in the core cloud:
{"Pressure": 867, "TIMESTAMP": "2020-12-26T16:20:58-25", "Humidity": 49, "Temperature": 71.6, "Device EUI": "0004A30B001BAAAA"} sent!
Verify that the IoT message count is increasing in Azure Portal for the IoT Edge: