Introduction

Integrated Integrated Edge Cloud(IEC) is an Akraino approved blueprint family and part of Akraino Edge Stack, which intends to develop a fully integrated edge infrastructure solution, and the project is completely focused

towards Edge Computing. This open source software stack provides critical infrastructure to enable high performance, reduce latency, improve availability, lower operational overhead, provide scalability, address

security needs, and improve fault management.

The The first step test mainly focus on the Android system running on edge ARM Cloud environment and make sure the Android system available.

...

Testing Working Group Resources

Test Architecture

Image Added

As picture aboved show, an android phone (Client) connect to our Edge Android Cloud. We plan to test the functional completeness and performance:

application operation on the client side by adb
deploy robox by k8s
system performace monitor by prometheus

Image Removed

Test Bed

The testbed setup is shown in the below diagram.

figure1

Image Removed

figure2

Image Added

figure1： Run Robox Through K8s

Image Added

figure2: Collect Data From NodeImage Removed

Test Environment

...

Step1: startup the k8s cluster by execute “deploy/compass/deployIEC.sh”

cd iec && bash deploy/compass/deployIEC.sh

Step2: mount the robox image and start the session_manager

ssh robox@192.168.10.66 bash iec/src/foundation/scripts/robox/loadimages.sh
ssh robox@192.168.10.66 bash iec/src/foundation/scripts/robox/sm_ctrl.sh

Step3: run some components for Cluster

ssh robox@192.168.10.66 bash iec/src/foundation/scripts/robox/ load_components.sh

Step4: run robox by K8S

bash iec/src/foundation/scripts/robox/test_robox.sh a

Display data through different components

1. By kubectl

kubectl get node,pods,svc -o wide -n kube-system -n default

NAME      STATUS   ROLESAGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION
node/master Ready control-plane,master 2d14h v1.21.5 192.168.10.66 <none>      Ubuntu 18.04.6 LTS4.15.18          docker://20.10.14
node/work    Ready    <none>          2d14h   v1.21.5   192.168.10.62   <none>     Ubuntu 18.04.3 LTS   4.15.18          docker://20.10.14

NAME    READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
pod/anbox-6c447fbd7-k8zl7   1/1     Running   0      76s     10.244.1.4   work     <none>           <none>
pod/nginx-c9zr9               1/1     Running   0    2d14h   10.244.1.3   work     <none>           <none>

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE     SELECTOR
service/anbox        NodePort       172.16.1.74    <none>        8888:31000/TCP          76s     app=android
service/kubernetes   ClusterIP   172.16.1.1     <none>        443/TCP          2d14h   <none>
service/nginx        NodePort   172.16.1.138   <none>        80:31778/TCP     2d14h   app=nginx

2. By Kuboard

Do as the install documentation, then deploy the kuboard

kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3-swr.yaml

login web browser, then switch to the Pods tab, it display as follow:

Image Added

Image Removed

3. By Prometheus

login login web browser, then switch to the Targets tab, it display as follow:

Image Added

Image Removed

Then switch to the “Status” tab, the box is the event what we want to query, and type “perf_sched_sched_migrate_task”, After a few minutes, you can see

the monitoring curve as bellow:

Image RemovedImage Added

4. By grafana

step1: add data source

url: http://localhost:3000
user: admin
password: admin
Click the below on dashboard：
Setting->Add Data Sources->Add data source
Select prometheus, where URL http://192.168.10.62:9090 or http://localhost:9090， Click save&test.

step2: import data source

...

Click the below on dashboard：
+ -> import -> enter dashboardid 1860(prometheus node is 1860)
Just click load.

Image Added Image Removed

Display Data Through Grafana

Test API description

The test is to evaluate the Android container available.

...

To increase the workload, we need to start more robox instances.

Image RemovedImage Added

Download bomb squad apk on googleplay and install via adb

adb connect nodeip:port
adb install zdxfd.apk

...

2. Start the bomb squad and do the settings

Click on the application icon, and then make the relevant settings, such asas Auto test mode、Show FPS.

Image RemovedImage Removed

Image Added Image Added

We We ran up to 20 instances on a single node, each instance ran an auto-tested bomb squad app, and we could watch the realtime fps.

...

When the node is running at full load, you can collect the corresponding system events through perf, and execute the following commands to collect different events.

...

sudo perf stat -e cycles,instructions,cache-references,cache-misses,bus-cycles -a sleep 10

...

sudo perf stat -e dTLB-load-misses,l1d_tlb_refill,iTLB-load-misses,l1i_tlb_refill sleep 10

...

sudo perf sched latency --sort runtime

sudo perf sched latency --sort switch

4. Generate flame graph

We save the events collected by perf locally, and then generate a flame graph, so that we can more intuitively see the system performance bottleneck.

sudo perf record -e cache-misses -ag -- sleep 10
sudo perf script -i perf.data |../FlameGraph/stackcollapse-perf.pl > out.perf-folded
cat out.perf-folded | ../FlameGraph/flamegraph.pl > perf_cache.svg

sudo perf record -e probe_libc:malloc -agR sleep 10
sudo perf script -i perf.data |../FlameGraph/stackcollapse-perf.pl > out.perf-folded
cat out.perf-folded | ../FlameGraph/flamegraph.pl > perf_malloc.svg

The flame graph display effect is as follows

Image RemovedImage Added

Blueprint extension tests

The Test inputs

Test	Description	Result	Reference
Robox Image	Robox Image Auto build	PASS	Installation Doc
Robox Startup	Start Robox By Script	PASS	Installation Doc
App Installation	Install and run app on a robox	PASS

Test

Current Doc
Edge Cluster Deployment	Deploy Edge K8S Clusters	Pass	Installation Doc
Deploy Robox	Run Robox By K8s	PASS	Installation Doc
Deploy Prometheus	Run Prometheus on Work node	PASS	Installation Doc
Deploy Grafana	Run Grafana on Work node	PASS	Installation Doc
Perf events	Run Perf Background to Fetch System Events	PASS

Test

Current Doc
Flame graph	Generate Flame Graph	PASS

Test

Current Doc

...

BluVal Tests

The Test inputs

...

Bluval User Guide

Test Procedure

Clone BluVal Validation Framework into a arm64 Server:
Copy .kube/config file and SSH key to the Test Machine
Configure validation environment:

...

cat validation/bluval/bluval-robox.yaml

blueprint:
    name: robox
    layers:
        - os
        - docker
        - k8s
    # Any hardware some basic tests
    os: &os_robox
        -
            name: ltp
            what: ltp
            optional: "True"
        -
            name: cyclictest
            what: cyclictest
            optional: "True"
        -
            name: lynis
            what: lynis
            optional: "False"
        -
            name: vuls
            what: vuls
            optional: "False"
    docker: &docker_base
        -
            name: docker_bench
            what: docker_bench
            optional: "True"
    k8s: &k8s
        -
            name: conformance
            what: conformance
            optional: "False"
        -
            name: etcd_ha
            what: etcd_ha
            optional: "True"
        -
            name: kube-hunter
            what: kube-hunter
            optional: "False"

cat validation/bluval/volumes.yaml

volumes:
    ssh_key_dir:
        local: '/root/.ssh'
        target: '/root/.ssh'
    kube_config_dir:
        local: '/root/.kube/'
        target: '/root/.kube/'
    custom_variables_file:
        local: '/opt/akraino/validation/tests/variables.yaml'
        target: '/opt/akraino/validation/tests/variables.yaml'
    blueprint_dir:
        local: '/opt/akraino/validation/bluval'
        target: '/opt/akraino/validation/bluval'
    results_dir:
        local: '/opt/akraino/results'
        target: '/opt/akraino/results'
    openrc:
        local: ''
        target: '/root/openrc'
layers:
    common:
        - custom_variables_file
        - blueprint_dir
        - results_dir
    hardware:
        - ssh_key_dir
    os:
        - ssh_key_dir
    networking:
        - ssh_key_dir
    docker:
        - ssh_key_dir
    k8s:
        - ssh_key_dir
        - kube_config_dir
    k8s_networking:
        - ssh_key_dir
        - kube_config_dir
    openstack:
        - openrc
    sds:
    sdn:
    vim:

cat validation/tests/variables.yaml

### Input variables cluster's master host
host: 192.168.10.66             # cluster's master host address
username: robox            # login name to connect to cluster
password: 123456         # login password to connect to cluster
ssh_keyfile: /root/.ssh/id_rsa # Identity file for authentication

Run BluVal Robot:

bash validation/bluval/blucon.sh robox

Install LFTOOLS:

sudo apt install python3-pip
sudo python3 -m pip install -U pip
sudo python3 -m pip install -U setuptools
sudo -H pip3 install --ignore-installed PyYAML

...

pip3 install lftools

2. Push BluVal Results to Akraino Nexus

...

# Create .netrc file
vi ~/.netrc
machine nexus.akraino.org login ysemicn password xxx

# Archive log files
zip -r results.zip ./results

# Push logs to Nexus
lftools deploy nexus-zip https://nexus.akraino.org logs ysemi/job/v1 results.zip

Expected output:

...

Loading KWallet
Loading SecretService
Loading Windows
Loading chainer
Loading macOS
Zip file upload complete.

...

Test Results

https://nexus.akraino.org/content/sites/logs/ysemi/job/v1/ak_results/

Vuls

Image Added

Image Removed

Lynis

Image RemovedImage Added

K8S Conformance

Image Removed

Image Added

Kube-Hunter

Image Added

Image Removed

Test Dashboards

...

Space shortcuts

Page tree

Versions Compared

Old Version 2

New Version 3

Key

Table of Contents

Introduction

Test Architecture

Test Bed

Test Environment

Display data through different components

1. By kubectl

2. By Kuboard

3. By Prometheus

4. By grafana

step1: add data source

step2: import data source

2. Start the bomb squad and do the settings

4. Generate flame graph

Blueprint extension tests

BluVal Tests

The Test inputs

Test Procedure

Test Results

Test Dashboards

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 2

New Version 3

Key

Table of Contents

Introduction

Test Architecture

Test Bed

Test Environment

Display data through different components

1. By kubectl

2. By Kuboard

3. By Prometheus

4. By grafana

step1: add data source

step2: import data source

2. Start the bomb squad and do the settings

4. Generate flame graph

Blueprint extension tests

BluVal Tests

The Test inputs

Test Procedure

Test Results

Test Dashboards