Table of Contents |
---|
Introduction
IEC Type3 mainly focus on Android Application running on edge ARM Cloud architecture with GPU/ vGPU Management.
...
Also, ARM cloud games need to have the basic features of "cloud”, such as flexibility , availability everywhere. Based on cloud infrastructure optimized for android application, providing ARM application services.
How to use this document
This document describes the construction, compilation and use of robox Android container environment. Introduce environment configuration, code download, build and compile, and how to use it.
This document is mainly suitable for users who build and compile robox container Android emulation.
Deployment Architecture
Pre-Installation Requirements
- Hardware Requirements
12*x86 server: The remaining memory is greater than 100G, and JAVA1.8.X is pre-installed.1*arm server:arm64 server:
Arch | AARCH64 |
Processor model | 1* |
Aarch64 processor( A72 inside) | |
RAM | 16*DDR4-2933 |
Storage | 10*2.5 inch SAS/SATA/SSD or 8*2.5 inch NVMe SSD |
Network | 1 onboard network card, each card supports 4*GE port or 4*10GE port or 4*25GE port |
Power Supply | Power 100~240V AC,240V DC |
Scale | 447 mm x 490 mm x 86.1 mm |
ARM Server satisfies the Arm Server Ready certified.
- Software Perequisites
item | method | |
os | ubuntu 18.04.3(key) |
|
robox | robox is an Android container. |
1. Log in to the server via SSH and enter the command line execution entry.
2. Enter a directory where the downloaded source code package is stored, such as the "/home/ubuntu" directory.
cd /home/ubuntu
3. Download the robox source code package and switch to the release-phase2.3 branch.
git clone https://github.com/ |
-b release-phase2.3 |
robox |
If the source code download fails, use the following command to skip verification:
export GIT_SSL_NO_VERIFY=1 )
roboxCompile and run dependent packages |
support for robox compilation and operation |
. |
apt-get install build-essential cmake cmake-data debhelper dbus google-mock libboost-dev libboost-filesystem-dev libboost-log-dev libboost-iostreams-dev libboost-program-options-dev libboost-system-dev libboost-test-dev libboost-thread-dev libcap-dev libdbus-1-dev libdbus-cpp-dev libegl1-mesa-dev libgles2-mesa-dev libglib2.0-dev libglm-dev libgtest-dev liblxc1 libproperties-cpp-dev libprotobuf-dev libsdl2-dev libsdl2-image-dev lxc-dev pkg-config protobuf-compiler |
docker run dependent packages
The installation is complete before configuring docker.
- Log in to the server remotely via SSH and enter the command line execution portal.
- Download and install the docker dependency package.
docker | needed by K8S/Robox | apt-get install docker.io |
- Components Version
Anbox | Run Android applications on any GNU/Linux operating system. | |
Grafana | Compose and scale observability with one or all pieces of the stack | 8.4.3 |
Prometheus | Cloud native system performance monitoring | 2.34.0 |
K8s | container orchestration engine for automating deployment, scaling, and management of containerized applications | k8s: v1.23.5; kube-apiserver:v1.21.11 kube-scheduler:v1.21.11 kube-proxy:v1.21.11 etcd:3.4.13-0 coredns:v1.8.0 |
- Database Perequisites
schema scripts: N/A
- Other Installation Requirements
- Jump Host Requirements:N/A
- Network Requirements: The network connection is normal and can communicate with the external network.
- Bare Metal Node Requirements :N/A
- Execution Requirements (Bare Metal Only) :N/A
Installation High-Level Overview
Bare Metal Deployment Guide
- Install Bare Metal Jump Host : N/A
- Creating a Node Inventory File: N/A
- Creating the Settings Files: N/A
- Running: N/AVirtual
Upstream Deployment Guide
- Install Ubuntu18.04.1 OS on arm Server
Operating system iso download link: http://old-releases.ubuntu.com/releases/18.04.1/ubuntu-18.04.1-server-arm64.iso
About how to install Ubuntu 18.04.1 on kunpeng arm server: https://support.huawei.com/enterprise/zh/doc/EDOC1100100619
...
- Local tool requirements
- 1. Install remote tools such as MobaXterm. MobaXterm is recommended.
- 2. Install VNC Viewer. Download link: https://www.realvnc.com/en/connect/download/viewer/.
...
Execute command modification.
sed -i "s/# deb-src/ deb-src/g" /etc/apt/sources.list
After modification as follows:
3. Update source.
apt-get update
-
- Install dependent libraries and basic components
1. Remove the comment of deb-src in /etc/apt/source.list.
sed -i "s/# deb-src/ deb-src/g" /etc/apt/sources.list apt update
2. Install the basic system dependency library.
Notice:
If there is a failure to obtain the deb package during the installation process, please download and install manually according to the URL in the prompt, and then reinstall the unsuccessful package.
apt install dpkg libncurses5-dev libncursesw5-dev libssl-dev cmake cmake-data debhelper dbus google-mock libboost-dev libboost-filesystem-dev libboost-log-dev libboost-iostreams-dev libboost-program-options-dev libboost-system- dev libboost-test-dev libboost-thread-dev libcap-dev libdbus-1-dev libegl1-mesa-dev libgles2-mesa-dev libglib2.0-dev libglm-dev libgtest-dev liblxc1 libproperties-cpp-dev libprotobuf-dev libsdl2 -dev libsdl2-image-dev lxc-dev pkg-config protobuf-compiler libboost-filesystem1.62.0 libboost-system1.62.0 docker.io dkms libboost-iostreams1.62.0 apt install build-essential apt install mesa-common-dev
3. Download location https://launchpad.net/ubuntu/zesty/arm64/libprocess-cpp3/3.0.1-0ubuntu5.
dpkg -i libprocess-cpp3_3.0.1-0ubuntu5_arm64.deb
4. The missing libdbus-cpp5_5.0.0+16.10.20160809-0ubuntu2_arm64.deb dependent library is downloaded from https://launchpad.net/ubuntu/zesty/arm64/libdbus-cpp5/5.0.0+16.10.20160809-0ubuntu2.
dpkg -i libdbus-cpp5_5.0.0+16.10.20160809-0ubuntu2_arm64.deb
5. The missing libdbus-cpp-dev_5.0.0+16.10.20160809-0ubuntu2_arm64.deb dependent library from https://launchpad.net/ubuntu/bionic/arm64/libdbus-cpp-dev/5.0.0+16.10.20160809- 0ubuntu2 official download.
dpkg -i libdbus-cpp-dev_5.0.0+16.10.20160809-0ubuntu2_arm64.deb
...
Open a new window and perform the following operations.
1. Log in to the server via SSH and enter the command line execution entry.
2. View docker information.
docker info
...
1) Open the "/etc/default/docker" file.
vim /etc/default/docker
- Upstream Deployment Key Features :N/A
- Special Requirements for Upstream Deployments :N/A
- Scenarios and Deploy Settings for Upstream Deployments :N/A
- Including Upstream Patches with Deployment:N/A
- Running: N/A
- Interacting with Containerized Overcloud: N/A
Verifying the Setup as defined the Akraino validation feature project plus any additional testing specific to the blue print
Install Main Components
Since the components and images required by the project are relatively large, and the process of compilation takes time, we store the compiled images on the github repository. The link is: https://github.com/ysemi-computing/RoboxWidget.git
git clone https://github.com/ysemi-computing/RoboxWidget.git
The code structure after downloading is as follows:
RoboxWidget/ ├── android.img ├── components │ ├── grafana-server │ ├── node_exporter │ ├── perf_exporter │ └── prometheus └── README.md |
Run Anbox On Ubuntu18.04
Build Android image for anbox
Code cloning and compilation is very time-consuming. If you just want to deploy and experience the Android system by iec, you can skip this section, because the project you just cloned already exists.
Firstly, you should install a ubuntu 18.04 system on Aarch64 processor, and swith to root user, then run the script “build_android_image.sh”。
After about two hours, the Android image was compiled successfully, and the results are as follows:
ls out/target/product/arm64/ android-info.txt obj previous_build_config.mk recovery symbols system.img build_fingerprint.txt cache.img data gen module-info.json ramdisk.img root system userdata.img |
Required packages installation
Before running anbox, we need to install some necessary packages, you can execute the fol script “arm64_env_setup.sh”
After this, the image of anbox has been completed, which can be viewed through docker related commands。Type the command below:
sudo docker images | grep robox android robox e223a91c4b58 6 days ago 860MB |
Run robox android system
After the above two scripts are executed, the robox operating environment is ready, and the entire directory structure of robox can be seen as follows
├── android ├── binaryFiles ├── cmake ├── COPYING.GPL ├── cross-compile-chroot.sh ├── data ├── docs ├── external ├── kernel ├── patch ├── products ├── scripts ├── src |
Step1: You can copy the binaryFiles in the robox code to the home directory, then run robox
start the first continer instance
./robox -v start 1
start the second continer instance
./robox -v start 2
Step2: Check the docker instance process.
sudo docker ps | grep instance
step3: Check the anbox session manager process
ps -aux | grep session
step4: Log in the container and confirm whether the robox instance started successfully.
docker exec -it instance2 sh
step5: get android property sys.boot_completed
getprop | grep sys.boot.completed
[sys.boot_completed]: [1] //1 which means start successfully
Cluster deployment and access by K8S
It is troublesome to start anbox through commands to deploy in a real environment. Here, the well-known K8S system is used to complete the deployment of anbox instances. On this basis, anbox can be much more effectively deployed, run, monitored, and analyzed for multiple host nodes. Here is k8s cluster setup and container orchestration.
1 Basic Architecture
2 K8s Cluster construction
The k8s cluster construction can be completed through the warehouse code, you can do as follows
git clone https://gerrit.akraino.org/r/iec.git cd iec/deploy/compass && bash deployIEC.sh
|
Here is the current configuration:
l Ubuntu Version:18.04 l Docker: 20+ l k8s: 1.21.3
|
l Host Network: master: 192.168.10.66 work: 192.168.10.62 |
3 Check cluster running status
kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health":"true"} kubectl get node NAME STATUS ROLES AGE VERSION master Ready control-plane,master 22h v1.21.3 work Ready <none> 22h v1.21.5 At the same time, we can present the status of the cluster through the k8s visual component. Here we use kuboard, which is very convenient to install and use. |
kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3-swr.yaml
then start kuboard with web browser
url:master-ip-address:30080
user:admin
password:Kuboard123
Cloud platform monitoring & Analyze
1 Basic Architecture
2 Bootup basic components
Prometheus is only used for monitoring data. The real data needs an exporter. Different exporters export different data,These data are finally presented by prometheus. Only node exporter and perf exporter are used in this project, these two can be directly downloaded and installed with binary files.
Prometheus is used for data monitoring, and then a better visual interface is grafana, the data it needs can just be obtained through Prometheus.
This project only needs to run the script below
https://github.com/ysemi-computing/RoboxWidget/blob/main/scripts/load_components.sh
.
After the script is executed, the required Prometheus and grafana are downloaded and run in the background
3 Analyze data by web browser
Let us explore data that Prometheus has collected about itself. To use Prometheus's built-in expression browser, navigate to http://localhost:9090/graph and choose the "Table" view within the "Graph" tab. Enter the below into the expression console and then click "Execute":
prometheus_target_interval_length_seconds
In addition, we can log in to grafana through a web browser and see the effect as shown below
user: admin
password: admin
2) Add the following script.
DOCKER_OPTS= -s overlay
3)It will take effect after restarting docker or reboot.
/etc/init.d/docker restart
-
- Install Remote Desktop
Step 1 Install remote desktop.
...
After that, you can access the graphical desktop of the server where the Huawei Kunpeng 920 processor is located through the Windows remote desktop, and log in with the root account.
Incorporate Exagear transcoding patch
...
1. Create a directory.
mkdir -p /home/compiler
2. Download the three source files of linux_4.15.0.orig.tar.gz, linux_4.15.0-65.74.diff.gz, linux_4.15.0-65.74.dsc, and put them into the "/home/compiler/" directory.
...
3. Enter the "/home/compiler/" directory.
cd /home/compiler/
...
linux_4.15.0-65.74.diff.gz linux_4.15.0-65.74.dsc linux_4.15.0.orig.tar.gz
...
4. The source code directory linux-4.15.0 will be generated in the current directory.
...
dpkg-source -x linux_4.15.0-65.74.dsc
5. Please extract exagear-a32a64-docker.tar.gz to the /home/ directory, and rename the decompressed folder to exagear.
...
tar zxvf exagear-a32a64-docker.tar.gz
mv "ExaGear ARM32-ARM64 for Android Docker" exagear //Rename the purpose: to facilitate subsequent use
cd /home/exagear
ls
android //安卓源码转码补丁
translator //物理机上用于转码的二进制可执行文件
kernel //物理机内核的转码补丁
...
6. Copy the transcoded patch to the kernel source directory.
cp /home/exagear/kernel/ubuntu-4.15.0-65.74.patch /home/compiler/linux-4.15.0/
cd /home/compiler/linux-4.15.0/
patch -p1 <ubuntu-4.15.0-65.74.patch
- Compile and install the kernel
1. Save and exit directly, and generate the configuration file .config vim .config.
make menuconfig
2. Confirm the following configuration items, if none or inconsistent, please manually change or add.
CONFIG_EXAGEAR_BT=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_PROC_CHILDREN=y
CONFIG_VFAT_FS=y
CONFIG_INPUT_UINPUT=y
3. Compile and install.
make -j64 //compile
make modules_install //Install modules
make install // install the kernel
cd /boot/grub
sudo update-grub2 //Update startup items
4. Restart.
reboot
-
- Transcode registration after the new kernel is enabled
1. After the physical machine restarts, the transcoding registration is enabled.
2. Mount the binfmt_misc file system, which is mounted by default. If it is not mounted, please execute it manually.
mount -t binfmt_misc none /proc/sys/fs/binfmt_misc3. Register exagear transcoding rules, pay attention to the following multiple places to store and use the "/opt/exagear/ubt_a32a64" path information of the ubt_a32a64 transcoding binary to be consistent.
mkdir -p /opt/exagearcp /home/exagear/translator/ubt_a32a64 /opt/exagear/
cd /opt/exagear //The directory for storing transcoded binary files. When registering echo, please pay attention to /opt/exagear/ubt_a32a64
chmod +x ubt_a32a64 //Avoid registration permission denied
- 4. Check whether the exagear rules are successfully registered, and ensure that the "/opt/exagear/ubt_a32a64" path information is consistent.
cat /proc/sys/fs/binfmt_misc/ubt_a32a64
-
- Compile and install binder.ko and ashmem.ko modules
1. Download the kernel source code for the kernel module to compile.
apt search linux-source apt install linux-source-4.15.0
2. Download the code and roll back to the specified version. Community link https://github.com/anbox/anbox-modules.
cd /home git clone https://github.com/anbox/anbox-modules.git
cd anbox-modules/ git reset --hard 816dd4d6e702cf77a44cfe208659af6c39e02b57
3. update ashmem and binder.
cd /home/anbox-modules
cp anbox.conf /etc/modules-load.d/
cp 99-anbox.rules /lib/udev/rules.d/
cp -rT ashmem /usr/src/anbox-ashmem-1
cp -rT binder /usr/src/anbox-binder-1
4. Compile and install using dkms.
dkms install anbox-ashmem/1
dkms install anbox-binder/1
5. Install the ko module to the kernel, and the binder_linux module needs to take parameters.
modprobe ashmem_linux
modprobe binder_linux num_devices=254
lsmod | grep -e ashmem_linux -e binder_linux
6. if the attribute permissions of ashmem and binder are not displayed by the following commands, please add the permissions with chmod.
ls -alh /dev/binder* /dev/ashmem
- Compile and install binder.ko and ashmem.ko modules
-
- Package it through android.img and store it in docker
1. Log in to the server by SSH and enter the command line execution portal.
2. Mount the android image to the "/mnt" directory.
cd /home mount /home/android.img /mnt
3. CD /mnt
4. image register
tar --numeric-owner -cf- . | docker import - android:robox_with_exagear
5. inquire android container
docker images
- Compile Robox source code
...
1. create compile directory
cd /home/robox-master
mkdir build
cd build
2. Compile the configuration and execute the command in the "/home/robox-master/build" directory.
cmake ..
3. Add the following in the "/usr/include/glm/gtx/transform.hpp" file.
vim /usr/include/glm/gtx/transform.hpp
...
Add the following definition at line 21:
#define GLM_ENABLE_EXPERIMENTAL
4. compile and install
make j64
- make install
...
- Start robox Android system
1.Copy the startup script robox in the binaryFiles directory in the downloaded robox-master source code to /home/robox-master
2. Execute the modified robox script and start the robox Android container
a. Set environment variables and assign containers to GPU cards to run.
export DISPLAY=:0
start the first continer instance
./robox -v start 1
start the second continer instance
./robox -v start 2
3.Check the docker instance process.
docker ps
4.Check the host session process, mainly check whether the two sessions corresponding to instance1 and instance2 are running normally.
ps -aux | grep session
5.Log in to the Android container and confirm whether the robox instance started successfully.
docker exec -it instance1 sh
77b2c041315f:/ # getprop | grep sys.boot.completed
[sys.boot_completed]: [1]
//sys.boot.completed shows 1 which means start successfully
Upstream Deployment Guide
- Upstream Deployment Key Features :N/A
- Special Requirements for Upstream Deployments :N/A
- Scenarios and Deploy Settings for Upstream Deployments :N/A
- Including Upstream Patches with Deployment:N/A
- Running: N/A
- Interacting with Containerized Overcloud: N/A
Verifying the Setup as defined the Akraino validation feature project plus any additional testing specific to the blue print
Developer Guide and Troubleshooting
- Utilization of Images
N/A
- Post-deployment Configuration
N/A
- Debugging Failures
N/A
- Reporting a Bug
N/A
Uninstall Guide
N/A
Troubleshooting
- If kernels modules successfully loaded, service is running fine, and yet Anbox won't launch with
Application manager service is not running yet
. Check for the messageNo space left on device - Failed to enter cgroup
in/var/lib/anbox/logs/container.log
, you may be running into see this issue: https://github.com/lxc/lxd/issues/6257 - If container started failed
Before the container is started, session-manager needs to be started on the host to start x, input, adb and other related services, and then the container may started successful.
To build the Anbox runtime itself there is nothing special to know. but when we use cmake as build system. A few build dependencies need to be present on your host system:
When the server restarts, the binder_linux module needs to be loaded and installed as follows:
sudo modprobe ashmem_linux sudo rmmod binder_linux || true; sudo modprobe binder_linux num_devices=254 sudo chmod 777 /dev/ashmem /dev/binder* |
If there is an external graphics card, you need to manually disable the built-in graphics card after each server restart. for example:
lspci | grep -in VGA 6:0007:41:00.0 VGA compatible controller: Huawei Technologies Co., Ltd. Hi1710 [iBMC Intelligent Management system chip w/VGA support] (rev 01) echo 1 > /sys/bus/pci/devices/0007\:41\:00.0/remove |
...
Maintenance
- Blue Print Package Maintenance
- Software maintenance: N/A
- Hardware maintenance:N/A
- Blue Print Deployment Maintenance (N/A)
Frequently Asked Questions
N/A
License
N/A
References
N/A
Definitions, acronyms and abbreviations
N/A
When porting an Android app to Anbox Cloud (usually in the form of an APK), there are a few issues that might cause your app to not function properly: