Table of Contents
Introduction
Integrated Edge Cloud(IEC) is is an Akraino approved blueprint family and part of Akraino Edge Stack, which intends to develop a fully integrated edge infrastructure solution, and the project is completely focused towards Edge Computing. This open source software stack provides critical infrastructure to enable high performance, reduce latency, improve availability, lower operational overhead, provide scalability, address security needs, and improve fault management.
The first step test mainly focus on the Android system running on edge ARM Cloud environment and make sure the Android system available.
Akraino Test Group Information
Testing Working Group Resources
Test Architecture
...
As picture aboved show, an android phone (Client) connect to our Edge Android Cloud. We plan to test test the functional completeness functional completeness and performance:
- application operation application operation on the client side
- latency fps
- by adb
- deploy robox by k8s
- system performace monitor by prometheus
Test Bed
The testbed setup is shown in the below diagram.
figure1
Test Framework
Running application and network monitors.
In this test process, there is nine video test sample to test the functional completeness and performance:
- application operation on the client side
- latency
- fps
The test environment we used:
- Hardware
figure2
Test Environment
Hardware Requirements
2*arm64 server:
Arch | Aarch64 |
Processor model | 1*Aarch64 processor, 1* Aarch64 processor, |
Processor model
RAM | 16*DDR4-2933 |
Storage | 10*2.5 inch SAS/SATA/SSD or 8*2.5 inch NVMe SSD |
Network | 1 onboard network card, each card supports 4*GE port or 4*10GE port or 4*25GE port |
Power Supply | Power 100~240V AC,240V DC |
Scale
447 mm x 490 mm x 86.1 mm
- Software
ARM Server satisfies the Arm Server Ready certified.
Software Perequisites
item | description | addition |
os | ubuntu 18.04.3(key) | |
robox | Android container |
docker | container for android image | apt-get install docker.io |
Components Version
Anbox | Run Android applications on any GNU/Linux operating system. | |
Grafana | Compose and scale observability with one or all pieces of the stack | 8.4.3 |
Prometheus | Cloud native system performance monitoring | 2.34.0 |
K8s | container orchestration engine for automating deployment, scaling, and management of containerized applications | k8s: v1.23.5; kube-apiserver:v1.21.11 kube-scheduler:v1.21.11 kube-proxy:v1.21.11 etcd:3.4.13-0 coredns:v1.8.0 |
Bootup Basic components
We have cloned the iec repository code to github, the link is:
...
...
...
If the source code download fails, use the following command to skip verification:
...
then do as follow steps, all operations are on master node.
Step1: startup the k8s cluster by execute “deploy/compass/deployIEC.sh”
cd iec && bash deploy/compass/deployIEC.sh
Step2: mount the robox image and start the session_manager
ssh robox@192.168.10.66 bash iec/src/foundation/scripts/robox/loadimages.sh
ssh robox@192.168.10.66 bash iec/src/foundation/scripts/robox/sm_ctrl.sh
Step3: run some components for Cluster
ssh robox@192.168.10.66 bash iec/src/foundation/scripts/robox/ load_components.sh
Step4: run robox by K8S
bash iec/src/foundation/scripts/robox/test_robox.sh a
Display data through different components
1. By kubectl
kubectl get node,pods,svc -o wide -n kube-system -n default
NAME STATUS ROLESAGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION node/master Ready control-plane,master 2d14h v1.21.5 192.168.10.66 <none> Ubuntu 18.04.6 LTS4.15.18 docker://20.10.14 node/work Ready <none> 2d14h v1.21.5 192.168.10.62 <none> Ubuntu 18.04.3 LTS 4.15.18 docker://20.10.14 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/anbox-6c447fbd7-k8zl7 1/1 Running 0 76s 10.244.1.4 work <none> <none> pod/nginx-c9zr9 1/1 Running 0 2d14h 10.244.1.3 work <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/anbox NodePort 172.16.1.74 <none> 8888:31000/TCP 76s app=android service/kubernetes ClusterIP 172.16.1.1 <none> 443/TCP 2d14h <none> service/nginx NodePort 172.16.1.138 <none> 80:31778/TCP 2d14h app=nginx |
2. By Kuboard
kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3-swr.yaml
login web browser, then switch to the Pods tab, it display as follow:
3. By Prometheus
login web browser, then switch to the Targets tab, it display as follow:
Then switch to the “Status” tab, the box is the event what we want to query, and type “perf_sched_sched_migrate_task”, After a few minutes, you can see the monitoring curve as bellow:
4. By grafana
step1: add data source
user: admin
password: admin
Click the below on dashboard:
Setting->Add Data Sources->Add data source
Select prometheus, where URL http://192.168.10.62:9090 or http://localhost:9090, Click save&test.
step2: import data source
Click the below on dashboard:
+ -> import -> enter dashboardid 1860(prometheus node is 1860)
Just click load.
...
Traffic Generator
Ping test
Test API description
The test is to evaluate the Android container available.
Thus we currently don't have any Test APIs provided.
Test Dashboards
Single pane view of how the test score looks like for the Blue print.
Node Proformance Test
1. Install bomb squad on every Robox
Since the instance runs on the server, we need to use the remote tool vnc, so that we can interact through GUI.
To increase the workload, we need to start more robox instances.
Download bomb squad apk on googleplay and install via adb
adb connect nodeip:port adb install zdxfd.apk |
2. Start the bomb squad and do the settings
Click on the application icon, and then make the relevant settings, such as
Auto test mode、Show FPS.
We ran up to 20 instances on a single node, each instance ran an auto-tested bomb squad app, and we could watch the realtime fps.
3. Run perf in the background for event collection
When the node is running at full load, you can collect the corresponding system events through perf, and execute the following commands to collect different events.
sudo perf stat -e cycles,instructions,cache-references,cache-misses,bus-cycles -a sleep 10
sudo perf stat -e dTLB-load-misses,l1d_tlb_refill,iTLB-load-misses,l1i_tlb_refill sleep 10
sudo perf sched latency --sort runtime
sudo perf sched latency --sort switch |
4. Generate flame graph
We save the events collected by perf locally, and then generate a flame graph, so that we can more intuitively see the system performance bottleneck.
sudo perf record -e cache-misses -ag -- sleep 10 sudo perf script -i perf.data |../FlameGraph/stackcollapse-perf.pl > out.perf-folded cat out.perf-folded | ../FlameGraph/flamegraph.pl > perf_cache.svg |
sudo perf record -e probe_libc:malloc -agR sleep 10 sudo perf script -i perf.data |../FlameGraph/stackcollapse-perf.pl > out.perf-folded cat out.perf-folded | ../FlameGraph/flamegraph.pl > perf_malloc.svg
|
The flame graph display effect is as follows
Blueprint extension tests
The Test inputs
Test | Description | Result | Reference |
Robox Image | Robox Image Auto build | PASS | Installation Doc |
Robox Startup | Start Robox By Script | PASS | Installation Doc |
App Installation | Install and run app on a robox | PASS | Test Doc |
Edge Cluster Deployment | Deploy Edge K8S Clusters | Pass | Installation Doc |
Deploy Robox | Run Robox By K8s | PASS | Installation Doc |
Deploy Prometheus | Run Prometheus on Work node | PASS | Installation Doc |
Deploy Grafana | Run Grafana on Work node | PASS | Installation Doc |
Perf events | Run Perf Background to Fetch System Events | PASS | Test Doc |
Flame graph | Generate Flame Graph | PASS | Test Doc |
BluVal Tests
The Test inputs
BluVal Test Environment setup according to:
Test Procedure
- Clone BluVal Validation Framework into a arm64 Server:
- Copy .kube/config file and SSH key to the Test Machine
- Configure validation environment:
cat validation/bluval/bluval-robox.yaml blueprint: name: robox layers: - os - docker - k8s # Any hardware some basic tests os: &os_robox - name: ltp what: ltp optional: "True" - name: cyclictest what: cyclictest optional: "True" - name: lynis what: lynis optional: "False" - name: vuls what: vuls optional: "False" docker: &docker_base - name: docker_bench what: docker_bench optional: "True" k8s: &k8s - name: conformance what: conformance optional: "False" - name: etcd_ha what: etcd_ha optional: "True" - name: kube-hunter what: kube-hunter optional: "False" |
cat validation/bluval/volumes.yaml volumes: ssh_key_dir: local: '/root/.ssh' target: '/root/.ssh' kube_config_dir: local: '/root/.kube/' target: '/root/.kube/' custom_variables_file: local: '/opt/akraino/validation/tests/variables.yaml' target: '/opt/akraino/validation/tests/variables.yaml' blueprint_dir: local: '/opt/akraino/validation/bluval' target: '/opt/akraino/validation/bluval' results_dir: local: '/opt/akraino/results' target: '/opt/akraino/results' openrc: local: '' target: '/root/openrc' layers: common: - custom_variables_file - blueprint_dir - results_dir hardware: - ssh_key_dir os: - ssh_key_dir networking: - ssh_key_dir docker: - ssh_key_dir k8s: - ssh_key_dir - kube_config_dir k8s_networking: - ssh_key_dir - kube_config_dir openstack: - openrc sds: sdn: vim: |
cat validation/tests/variables.yaml ### Input variables cluster's master host host: 192.168.10.66 # cluster's master host address username: robox # login name to connect to cluster password: 123456 # login password to connect to cluster ssh_keyfile: /root/.ssh/id_rsa # Identity file for authentication |
- Run BluVal Robot:
bash validation/bluval/blucon.sh robox
- Install LFTOOLS:
sudo apt install python3-pip sudo python3 -m pip install -U pip sudo python3 -m pip install -U setuptools sudo -H pip3 install --ignore-installed PyYAML
pip3 install lftools |
- Push BluVal Results to Akraino Nexus
# Create .netrc file vi ~/.netrc machine nexus.akraino.org login ysemicn password xxx
# Archive log files zip -r results.zip ./results
# Push logs to Nexus lftools deploy nexus-zip https://nexus.akraino.org logs ysemi/job/v1 results.zip |
Expected output:
Loading KWallet Loading SecretService Loading Windows Loading chainer Loading macOS Zip file upload complete. |
Test Results
https://nexus.akraino.org/content/sites/logs/ysemi/job/v1/ak_results/
Vuls
Lynis
K8S Conformance
Kube-Hunter
Test Dashboards
Single pane view of how the test score looks like for the Blue print.
Test Group | Total Tests | Pass | Fail |
Blueprint Extension Tests | 10 | 10 | 0 |
Vuls | 1 | 1 | 0 |
Lynis | 1 | 1 | 0 |
K8S Conformance | 1 | 0 | 1 |
Kube-Hunter | 1 | 1 | 0 |
Additional Testing
N/A
Bottlenecks/Errata
N/A