Table of Contents

Introduction

    IEC Type3 mainly focus on Android Application running on edge ARM Cloud architecture with GPU/ vGPU Management.

...

 Also, ARM cloud games need to have the basic features of "cloud”, such as flexibility ,

availability everywhere. Based on cloud infrastructure optimized for android application, providing ARM application services.

How to use this document

    This document describes the construction, compilation and use of robox Android container environment. Introduce environment configuration, code download, build and compile, and how to use it.

    This document is mainly suitable for users who build and compile robox container Android emulation.

Deployment Architecture

Image Removed

Pre-Installation Requirements

• Hardware Requirements

1*x86 server: The remaining memory is greater than 100G, and JAVA1.8.X is pre-installed.

1*arm server:

...

Processor model

...

1*Kunpeng  920 processor

...

RAM

...

16*DDR4-2933

...

Storage

...

10*2.5 inch SAS/SATA/SSD or 8*2.5 inch  NVMe SSD

...

Network

...

1 onboard network card, each card supports 4*GE port or 4*10GE port or 4*25GE port

...

Power Supply

...

 Power 100~240V AC，240V DC

...

Scale

...

447 mm x 490 mm x 86.1 mm

Image Added

                                                                                                                                                 Figure1     Deployment Main Framework

Image Added

                                                                                                                  Figure2    Robox Framework

Pre-Installation Requirements

• Hardware Requirements

2*arm64 server:

ARM Server satisfies the Arm Server Ready certified.

• Software Perequisites

ARM Server satisfies the Arm Server Ready certified.

• Software Perequisites

• Database Perequisites

schema scripts: N/A

• Other Installation Requirements

  • Jump Host Requirements：N/A
  • Network Requirements:  The network connection is normal and can communicate with the external network.
  • Bare Metal Node Requirements ：N/A
  • Execution Requirements (Bare Metal Only) ：N/A

Installation High-Level Overview

• Bare Metal Deployment Guide 

  • Install Bare Metal Jump Host : N/A

  • Creating a Node Inventory File: N/A

  • Creating the Settings Files: N/A

  • Running: N/A

• Virtual Deployment Guide

  • Install Ubuntu18.04.1 OS on arm Server

      Operating system iso download link: http://old-releases.ubuntu.com/releases/18.04.1/ubuntu-18.04.1-server-arm64.iso

      About how to install Ubuntu 18.04.1 on kunpeng arm server: https://support.huawei.com/enterprise/zh/doc/EDOC1100100619 

...

• Local tool requirements
• 1. Install remote tools such as MobaXterm. MobaXterm is recommended.
• 2. Install VNC Viewer. Download link: https://www.realvnc.com/en/connect/download/viewer/.

...

Execute command modification.

sed -i "s/# deb-src/ deb-src/g" /etc/apt/sources.list

After modification as follows:

Image Removed

3. Update source.

apt-get update

•  
  • Install dependent libraries and basic components

1. Remove the comment of deb-src in /etc/apt/source.list.

sed -i "s/# deb-src/ deb-src/g" /etc/apt/sources.list apt update

2. Install the basic system dependency library.
Notice:
If there is a failure to obtain the deb package during the installation process, please download and install manually according to the URL in the prompt, and then reinstall the unsuccessful package.

apt install dpkg libncurses5-dev libncursesw5-dev libssl-dev cmake cmake-data debhelper dbus google-mock libboost-dev libboost-filesystem-dev libboost-log-dev libboost-iostreams-dev libboost-program-options-dev libboost-system- dev libboost-test-dev libboost-thread-dev libcap-dev libdbus-1-dev libegl1-mesa-dev libgles2-mesa-dev libglib2.0-dev libglm-dev libgtest-dev liblxc1 libproperties-cpp-dev libprotobuf-dev libsdl2 -dev libsdl2-image-dev lxc-dev pkg-config protobuf-compiler libboost-filesystem1.62.0 libboost-system1.62.0 docker.io dkms libboost-iostreams1.62.0 apt install build-essential apt install mesa-common-dev

3. Download location https://launchpad.net/ubuntu/zesty/arm64/libprocess-cpp3/3.0.1-0ubuntu5.

dpkg -i libprocess-cpp3_3.0.1-0ubuntu5_arm64.deb

4. The missing libdbus-cpp5_5.0.0+16.10.20160809-0ubuntu2_arm64.deb dependent library is downloaded from https://launchpad.net/ubuntu/zesty/arm64/libdbus-cpp5/5.0.0+16.10.20160809-0ubuntu2.

dpkg -i libdbus-cpp5_5.0.0+16.10.20160809-0ubuntu2_arm64.deb

5. The missing libdbus-cpp-dev_5.0.0+16.10.20160809-0ubuntu2_arm64.deb dependent library from https://launchpad.net/ubuntu/bionic/arm64/libdbus-cpp-dev/5.0.0+16.10.20160809- 0ubuntu2 official download.

dpkg -i libdbus-cpp-dev_5.0.0+16.10.20160809-0ubuntu2_arm64.deb

...

Open a new window and perform the following operations.

1. Log in to the server via SSH and enter the command line execution entry.
2. View docker information.

• docker info

...

1) Open the "/etc/default/docker" file.

• vim /etc/default/docker

2) Add the following script.

• DOCKER_OPTS= -s overlay

3）It will take effect after restarting docker or reboot.

• • • /etc/init.d/docker restart

•  
  
  • Install Remote Desktop

Step 1 Install remote desktop.

...

After that, you can access the graphical desktop of the server where the Huawei Kunpeng 920 processor is located through the Windows remote desktop, and log in with the root account.

• Incorporate Exagear transcoding patch

...

1. Create a directory.
mkdir -p /home/compiler

2. Download the three source files of linux_4.15.0.orig.tar.gz, linux_4.15.0-65.74.diff.gz, linux_4.15.0-65.74.dsc, and put them into the "/home/compiler/" directory.

...

3. Enter the "/home/compiler/" directory.
cd /home/compiler/

...

• linux_4.15.0-65.74.diff.gz  linux_4.15.0-65.74.dsc  linux_4.15.0.orig.tar.gz

...

4. The source code directory linux-4.15.0 will be generated in the current directory.

...

dpkg-source -x linux_4.15.0-65.74.dsc

5. Please extract exagear-a32a64-docker.tar.gz to the /home/ directory, and rename the decompressed folder to exagear.

...

tar zxvf exagear-a32a64-docker.tar.gz

mv "ExaGear ARM32-ARM64 for Android Docker" exagear //Rename the purpose: to facilitate subsequent use

cd /home/exagear

ls

• android //安卓源码转码补丁

• translator //物理机上用于转码的二进制可执行文件

• kernel //物理机内核的转码补丁

...

6. Copy the transcoded patch to the kernel source directory.
cp /home/exagear/kernel/ubuntu-4.15.0-65.74.patch /home/compiler/linux-4.15.0/

cd /home/compiler/linux-4.15.0/

patch -p1 <ubuntu-4.15.0-65.74.patch

• Components Version

• Database Perequisites

schema scripts: N/A

• Other Installation Requirements

  • Jump Host Requirements：N/A

  • Network Requirements:  The network connection is normal and can communicate with the external network.

  • Bare Metal Node Requirements ：N/A

  • Execution Requirements (Bare Metal Only) ：N/A

Installation High-Level Overview

Bare Metal Deployment Guide 

• Install Bare Metal Jump Host : N/A

• Creating a Node Inventory File: N/A

• Creating the Settings Files: N/A

• Running: N/A

Upstream Deployment Guide

• Upstream Deployment Key Features ：N/A

• Special Requirements for Upstream Deployments ：N/A

• Scenarios and Deploy Settings for Upstream Deployments ：N/A

• Including Upstream Patches with Deployment：N/A

• Running： N/A

• Interacting with Containerized Overcloud： N/A

Verifying the Setup as defined the Akraino validation feature project plus any additional testing specific to the blue print

Install Main Components

       Since the components and images required by the project are relatively large, and the process of  compilation takes time, we store the compiled images on the github repository.

The link is: https://github.com/ysemi-computing/RoboxWidget.git

git clone https://github.com/ysemi-computing/RoboxWidget.git

After this step,  The code structure is as follows:

RoboxWidget/

├── android.img

├── components

├── grafana-server 

├── node_exporter

├── perf_exporter

└── prometheus

└── README.md

Run Anbox On Ubuntu18.04

Build Android image for anbox

        Code cloning and compilation is very time-consuming. If you just want to deploy and experience the Android system by iec, you can skip this section, because the project you just cloned already exists.

Firstly, you should install a ubuntu 18.04 system on Aarch64 processor, and swith to root user, then run the script “build_android_image.sh”。

After about two hours, the Android image was compiled successfully,  and the results are as follows:

ls out/target/product/arm64/

android-info.txt  obj   previous_build_config.mk  recovery  symbols  system.img  build_fingerprint.txt  cache.img  data  gen  module-info.json    ramdisk.img    root  system   userdata.img

Required packages installation

   Before running anbox, we need to install some necessary packages, you can execute the fol script “arm64_env_setup.sh”

After this,  the image of anbox has been completed, which can be viewed through docker related commands。Type the command below:

sudo docker images | grep robox

android                                                               robox                          e223a91c4b58   6 days ago      860MB

Run robox android system

After the above two scripts are executed, the robox operating environment is ready, and the entire directory structure of robox can be seen as follows

├── android

├── binaryFiles

├── cmake

├── cmake_uninstall.cmake.in

├── COPYING.GPL

├── cross-compile-chroot.sh

├── data

├── docs

├── external

├── kernel

├── patch

├── products

├── scripts

├── src

Step1: You can copy the binaryFiles in the robox code to the some place, then run robox

start the first continer instance

./robox -v start 1

       1 is the id, used by session manager and docker container

Step2: Check the docker instance process and session manager process.

sudo docker ps | grep instance

ps -aux | grep session

step3: Log in the container and confirm whether the robox instance started successfully.

   container name format:  instance + id

docker exec -it instance1 sh      

step4: get android property sys.boot_completed

getprop | grep sys.boot.completed

[sys.boot_completed]: [1]  //1 which means start successfully

Cluster deployment and access by K8S

        It is troublesome to start robox through commands to deploy in a real environment. Here, the well-known K8S system is used to complete the deployment of robox instances. On this basis,

Robox can be much more effectively deployed, run, monitored, and analyzed for multiple host nodes. Here is k8s cluster setup and container orchestration.

1 Basic Architecture

 Image Added

                                                                            Figure3    View Of Robox In Cluster

2 K8s Cluster construction

        The k8s cluster construction can be completed through the warehouse code, you can do as follows

git clone https://gerrit.akraino.org/r/iec.git

cd iec/deploy/compass && bash deployIEC.sh

current configuration:

• Ubuntu Version：18.04
• Docker: 20+
• k8s: 1.21.3

Host Network:

• master: 192.168.10.66
• work:   192.168.10.62

3 Check cluster running status

   kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3-swr.yaml

kubectl get cs 

Warning: v1 ComponentStatus is deprecated in v1.19+

NAME                 STATUS    MESSAGE             ERROR

scheduler            Healthy   ok                 

controller-manager   Healthy   ok                 

etcd-0               Healthy   {"health":"true"}  

kubectl get node

NAME     STATUS   ROLES                  AGE   VERSION

master   Ready    control-plane,master   22h   v1.21.3

work     Ready    <none>                 22h   v1.21.5

          At the same time, we can present the status of the cluster through the k8s visual component. We use kuboard, it is very convenient to install and use.

      then start kuboard with web browser

url：master-ip-address:30080

user：admin

password：Kuboard123

Image Added

                                                                                                         Figure4    View of Robox On Kuboard

Cloud platform monitoring & Analyze

1 Basic Architecture

Image Added

                                                                                                   Figure5    Prometheus and Grafana Mix

2 Bootup basic components

     Prometheus is only used for monitoring data. The real data needs an exporter. Different exporters export different data,These data are finally presented by prometheus.

Only node exporter and perf exporter are used in this project, these two can be directly downloaded and installed with binary files. Prometheus is used for data monitoring,

and then a better visual interface is grafana, the data it needs can just be obtained through Prometheus.

     This project only needs to run the script below

https://github.com/ysemi-computing/RoboxWidget/blob/main/scripts/load_components.sh

     After the script is executed, the required Prometheus and grafana are downloaded and run in the background

3 Analyze data by web browser

     Let us explore data that Prometheus has collected about itself. To use Prometheus's built-in expression browser, navigate to http://localhost:9090/graph and choose the "Table" view

within the "Graph" tab. Enter the below into the expression console and then click "Execute":

      prometheus_target_interval_length_seconds

Image Added

                                                                                                                   Figure6    First Startup View Of Prometheus

In addition, we can log in to grafana through a web browser and see the effect as shown below

url: http://localhost:3000

user: admin

password: admin

Image Added

                                                                                                                   Figure7     First Startup View Of Grafana

Developer Guide and Troubleshooting

• Utilization of Images

N/A

• Post-deployment Configuration

N/A

• Debugging Failures 

N/A

• Reporting a Bug

N/A

Uninstall Guide

N/A

Troubleshooting

When the server restarts, the binder_linux module needs to be loaded and installed as follows：

sudo modprobe ashmem_linux

sudo rmmod binder_linux || true; sudo modprobe binder_linux num_devices=254

sudo chmod 777 /dev/ashmem /dev/binder*

If there is an external graphics card, you need to manually disable the built-in graphics card after each server restart. for example:

lspci | grep -in VGA

6:0007:41:00.0 VGA compatible controller: Huawei Technologies Co., Ltd. Hi1710 [iBMC Intelligent Management system chip w/VGA support] (rev 01)

echo 1 > /sys/bus/pci/devices/0007\:41\:00.0/remove

• Compile and install the kernel

1. Save and exit directly, and generate the configuration file .config vim .config.
make menuconfig

2. Confirm the following configuration items, if none or inconsistent, please manually change or add.
CONFIG_EXAGEAR_BT=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_PROC_CHILDREN=y
CONFIG_VFAT_FS=y
CONFIG_INPUT_UINPUT=y

3. Compile and install.
make -j64 //compile

make modules_install //Install modules

make install // install the kernel

cd /boot/grub

sudo update-grub2 //Update startup items

4. Restart.
reboot

•  
  • Transcode registration after the new kernel is enabled

  • 1. After the physical machine restarts, the transcoding registration is enabled.
    2. Mount the binfmt_misc file system, which is mounted by default. If it is not mounted, please execute it manually.
    mount -t binfmt_misc none /proc/sys/fs/binfmt_misc

    3. Register exagear transcoding rules, pay attention to the following multiple places to store and use the "/opt/exagear/ubt_a32a64" path information of the ubt_a32a64 transcoding binary to be consistent.
    mkdir -p /opt/exagear

    cp /home/exagear/translator/ubt_a32a64 /opt/exagear/

    cd /opt/exagear //The directory for storing transcoded binary files. When registering echo, please pay attention to /opt/exagear/ubt_a32a64

    chmod +x ubt_a32a64 //Avoid registration permission denied

  •  4. Check whether the exagear rules are successfully registered, and ensure that the "/opt/exagear/ubt_a32a64" path information is consistent.
    cat /proc/sys/fs/binfmt_misc/ubt_a32a64

•  
  • Compile and install binder.ko and ashmem.ko modules

    1. Download the kernel source code for the kernel module to compile.

    apt search linux-source apt install linux-source-4.15.0 

    2. Download the code and roll back to the specified version. Community link https://github.com/anbox/anbox-modules.

    cd /home git clone https://github.com/anbox/anbox-modules.git

    cd anbox-modules/ git reset --hard 816dd4d6e702cf77a44cfe208659af6c39e02b57

    3. update ashmem and binder.

    cd /home/anbox-modules

    cp anbox.conf /etc/modules-load.d/

    cp 99-anbox.rules /lib/udev/rules.d/

    cp -rT ashmem /usr/src/anbox-ashmem-1

    cp -rT binder /usr/src/anbox-binder-1

    4. Compile and install using dkms.

    dkms install anbox-ashmem/1

    dkms install anbox-binder/1

    5. Install the ko module to the kernel, and the binder_linux module needs to take parameters.

    modprobe ashmem_linux

    modprobe binder_linux num_devices=254

    lsmod | grep -e ashmem_linux -e binder_linux

    6. if the attribute permissions of ashmem and binder are not displayed by the following commands, please add the permissions with chmod.

    ls -alh /dev/binder* /dev/ashmem

•  
  • Package it through android.img and store it in docker

1. Log in to the server by SSH and enter the command line execution portal.

2. Mount the android image to the "/mnt" directory.

cd /home mount /home/android.img /mnt

3. CD /mnt

4.  image register

tar --numeric-owner -cf- . | docker import - android:robox_with_exagear

5. inquire android  container

docker images

• Compile Robox source code

...

1. create compile directory

•               cd /home/robox-master

•               mkdir build

•               cd build

2. Compile the configuration and execute the command in the "/home/robox-master/build" directory.

cmake ..

3. Add the following in the "/usr/include/glm/gtx/transform.hpp" file.

•              vim /usr/include/glm/gtx/transform.hpp

...

Add the following definition at line 21: 

                        #define GLM_ENABLE_EXPERIMENTAL

4.  compile and install

make j64

•              make install

...

• Start robox Android system

                    1.Copy the startup script robox in the binaryFiles directory in the downloaded robox-master source code to /home/robox-master

2. Execute the modified robox script and start the robox Android container

a. Set environment variables and assign containers to GPU cards to run.

export DISPLAY=:0

start the first continer instance

./robox -v start 1

start the second continer instance

./robox -v start 2

3.Check the docker instance process.

docker ps

4.Check the host session process, mainly check whether the two sessions corresponding to instance1 and instance2 are running normally.

ps -aux | grep session

5.Log in to the Android container and confirm whether the robox instance started successfully.

docker exec -it instance1 sh

77b2c041315f:/ # getprop | grep sys.boot.completed

[sys.boot_completed]: [1]

//sys.boot.completed shows 1 which means start successfully

• Upstream Deployment Guide

  • Upstream Deployment Key Features ：N/A
  • Special Requirements for Upstream Deployments ：N/A
  • Scenarios and Deploy Settings for Upstream Deployments ：N/A
  • Including Upstream Patches with Deployment：N/A
  • Running： N/A
  • Interacting with Containerized Overcloud： N/A

Verifying the Setup as defined the Akraino validation feature project plus any additional testing specific to the blue print

Developer Guide and Troubleshooting

• Utilization of Images

N/A

• Post-deployment Configuration

N/A

• Debugging Failures 

N/A

• Reporting a Bug

N/A

Uninstall Guide

N/A

Troubleshooting

• If kernels modules successfully loaded, service is running fine, and yet Anbox won't launch with Application manager service is not running yet. Check for the message No space left on device - Failed to enter cgroup in /var/lib/anbox/logs/container.log, you may be running into see this issue: https://github.com/lxc/lxd/issues/6257
• If container started failed

Before the container is started, session-manager needs to be started on the host to start x, input, adb and other related services, and then the container may started successful.

To build the Anbox runtime itself there is nothing special to know. but when we use cmake as build system. A few build dependencies need to be present on your host system:

...

Maintenance

• Blue Print Package Maintenance

• Software maintenance： N/A 

• Hardware maintenance：N/A

• Blue Print Deployment Maintenance (N/A)

Frequently Asked Questions

N/A

License

N/A

References

N/A

Definitions, acronyms and abbreviations

N/A

When porting an Android app to Anbox Cloud (usually in the form of an APK), there are a few issues that might cause your app to not function properly: