Introduction
This document describes steps required to deploy a sample environment for the Public Cloud Edge Interface (PCEI) Blueprint.
Deployment Architecture
The deployment architecture is shown below. All Addressing and naming shown are for example purposes only.
Image Removed
Image Added
Deployment environment components:
- One Host Server
- One Edge Multi-Cluster Orchestrator VM
- Two Edge K8S Cluster VMs
- Internal network connectivity to the Host Server and to the VMs
Pre-Installation Requirements
Recommended Hardware Requirements
| Requirement | Value |
|---|
| CPU | 18 Core / 36 vCPU |
| RAM | 128 GB |
| DISK | 500 GB |
| NETWORK | 1 Gbps NIC Port |
Network Requirements
- Internal Lab Network (RFC 1918 space)
- Internet access (NAT'd)
- IP routing in the Lab Network to reach VM interfaces
Software Prerequisites
| Requirement | Value |
|---|
| Server OS | CentOS 7.x or above |
| VM OS | Ubuntu 18.04 |
| Upstream SW | Refer to Deployment Guide Section |
Installation High-Level Overview
The installation/deployment process consists of the following steps:
- Install CentOS on Host Server.
- Prepare Host Server for EMCO Deployment.
- Deploy EMCO.
- Deploy Edge Clusters.
Deployment Guide
Install CentOS on Host Server
- Connect to Host Server iLO interface.
- Start Virtual Console.
- Mount Virtual Media with CentOS 7 ISO.
- Install CentOS
- Assign correct IP address, Subnet, Gateway and DNS to the NIC.
- Include OpenSSH Server.
- Install KVM/virtualization.
- Add a user with admin privileges: onaplab user is used in this guide.
Prepare Host Server for EMCO Deployment
Step 1. Enable sudo without entering sudo password
| Code Block |
|---|
|
sudo -i
visudo
# Uncomment the line below
%wheel ALL=(ALL) NOPASSWD: ALL |
Step 2. Add sudo user to wheel group
| Code Block |
|---|
|
usermod –aG wheel onaplab |
Step 3. Enable nested virtualization
| Code Block |
|---|
|
# Login as super user
sudo -i
# Check for the following parameter
cat /sys/module/kvm_intel/parameters/nested
N
# If it is Y, there is nothing else to be done. It is already enabled!
# If it is N, do the following:
# Edit /etc/default/grub file, and include kvm-intel.nested=1
GRUB_CMDLINE_LINUX parameter
GRUB_CMDLINE_LINUX="crashkernel=auto
console=ttyS0,38400n8,kvm-intel.nested=1"
# Rebuild your GRUB configuration
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# Enable nested KVM capabilities in /etc/modprobe.d/kvm.conf
# By uncommenting the below line
options kvm_intel nested=1
### Reboot the server
reboot
# Now, we should have the nested KVM capabilities enabled
cat /sys/module/kvm_intel/parameters/nested
Y |
Step 4. Install VNC Server (Optional)
Follow instructions at:
https://www.tecmint.com/install-and-configure-vnc-server-in-centos-7/
Step 5. Modify libvirt bridge IP and route mode
This will allow connecting to VMs and pods directly from the Lab Network. Please replace the sample IPs with your IP addresses. Please replace the interface name (eno24 used in the example) with you server's interface name.
...
| Code Block |
|---|
|
ip route 10.121.7.128 255.255.255.224 10.121.1.12 |
Deploy EMCO
Step 1. Generate SSH Keys
| Code Block |
|---|
|
# Run commands below on the Host Server
ssh-keygen
cd ~/.ssh
chmod 600 id_rsa
chmod 600 id_rsa.pub
chmod 700 config
chmod 600 known_hosts
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys
echo "# Increase the server timeout value" >> ~/.ssh/config
echo "ServerAliveInterval 120" >> ~/.ssh/config |
Step 2. Download software and install EMCO
Note that the install process will:
...
| Code Block |
|---|
|
#### On the Host Server
sudo yum install -y git deltarpm
mkdir -p amcop_deploy
cd amcop_deploy
## Download the installation package zip file
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1PMyc8yULDeTIY0xNvY0RDf_7CSWvli3l' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1PMyc8yULDeTIY0xNvY0RDf_7CSWvli3l" -O amcop_install_v2.0.zip && rm -rf /tmp/cookies.txt
unzip amcop_install_v2.0.zip
sudo chown -R onaplab:onaplab ~/amcop_deploy/
cd ~/amcop_deploy/aarna-stream/util-scripts
./prep_baremetal_centos.sh
####### Install EMCO/AMCOP
# Edit inventory.ini file. Use IP address of Host Server and the username.
cd ~/amcop_deploy/aarna-stream/amcop_deploy/ansible/deployment
vi inventory.ini
[deployment_host]
10.121.1.12 ansible_user=onaplab
nohup ansible-playbook ./main.yml -i inventory.ini -e deployment_env=on-prem -e jump_host_user=onaplab --private-key=/home/onaplab/.ssh/id_rsa -e vm_user=onaplab & |
Step 3. Monitor the installation
| Code Block |
|---|
|
# On the Host Server
cd /home/onaplab/aarna-stream/anod_lite/logs
[onaplab@os12 logs]$ ls -l
total 1980
-rw-r--r--. 1 root root 510417 Nov 24 07:06 cluster_setup.log
-rw-r--r--. 1 root root 2019 Nov 24 06:54 create_vm.log
-rw-r--r--. 1 root root 1366779 Nov 24 07:15 deploy_emco_components.log
-rw-r--r--. 1 root root 138233 Nov 24 07:35 deploy_onap.log
-rw-rw-r--. 1 onaplab onaplab 83 Nov 24 06:53 README.md
tail -f create_vm.log
tail -f cluster_setup.log
tail -f deploy_emco_components.log
tail -f deploy_onap.log |
...
| Code Block |
|---|
|
sudo virsh destroy amcop-vm-01
sudo virsh undefine amcop-vm-01
sudo virsh pool-destroy amcop-vm-01
sudo virsh pool-undefine amcop-vm-01
sudo rm /var/lib/libvirt/images/amcop-vm-01/amcop-vm-01-cidata.iso
sudo rm /var/lib/libvirt/images/amcop-vm-01/amcop-vm-01.qcow2 |
Step 4. Install Controller Blueprint Archives (CBA)
Update CDS py-executor
https://gitlab.com/akraino-pcei-onap-cds/equinix-pcei-poc
Kubernetes Cluster Registration CBA
https://gitlab.com/akraino-pcei-onap-cds/equinix-pcei-poc/-/tree/main/cds-blueprints/emco-api-executor
https://gitlab.com/akraino-pcei-onap-cds/equinix-pcei-poc/-/tree/main/cds-blueprints/terraform-plan-executor
Helm Chart Processor CBA
https://gitlab.com/akraino-pcei-onap-cds/equinix-pcei-poc/-/tree/main/cds-blueprints/helm-chart-processor
Composite App Deployment Processor CBA
https://gitlab.com/akraino-pcei-onap-cds/equinix-pcei-poc/-/tree/main/cds-blueprints/composite-app-deployment-processor
Deploy Edge Clusters
Step 1. Edit VM creation script.
| Code Block |
|---|
|
# On the Host Server
cd /home/onaplab/amcop_deploy/aarna-stream/util-scripts
# Add "--cpu host" option to the end of the below line
vi create_qem_vm.sh
virt-install --connect qemu:///system --name $vm_name --ram $(($mem << 10)) --vcpus=$vCPU --os-type linux --os-variant $os_variant --disk path=/var/lib/libvirt/images/$vm_name/"$vm_name".qcow2,format=qcow2 --disk /var/lib/libvirt/images/$vm_name/$vm_name-cidata.iso,device=cdrom --import --network network=default --noautoconsole —-cpu host
# Save the file |
Step 2. Deploy two Edge Cluster VMs.
These commands will create two Ubuntu 18.04 VMs with 100G Disk, 8 vcpu and 16G RAM and will copy the contents of the ~/.ssh/id_rsa.pub key file from the Host Server to the VMs' ~/.ssh/authorized_keys file.
| Code Block |
|---|
|
sudo ./create_qem_vm.sh 2 edge_k8s-1 100 8 16 ubuntu18.04 $HOME/.ssh/id_rsa.pub onaplab
sudo ./create_qem_vm.sh 2 edge_k8s-2 100 8 16 ubuntu18.04 $HOME/.ssh/id_rsa.pub onaplab |
Step 3. Setup worker clusters inside VMs
| Code Block |
|---|
|
# Find VM's IP addresses. On the Host Server run:
[onaplab@os12 ~]$ sudo virsh list --all
Id Name State
----------------------------------------------------
6 amcop-vm-01 running
9 edge_k8s-1 running
10 edge_k8s-2 running
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-1
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet1 52:54:00:19:96:72 ipv4 10.121.7.152/27
[onaplab@os12 ~]$
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-2
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet2 52:54:00:c0:47:8b ipv4 10.121.7.146/27
# ssh to each VM from the Host Server:
ssh onaplab@10.121.7.152
ssh onaplab@10.121.7.146
# Perform the following tasks in each VM:
sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y python-pip
git clone https://git.onap.org/multicloud/k8s/
# Run script to setup KUD clusters
nohup k8s/kud/hosting_providers/baremetal/aio.sh %
|
...
| Code Block |
|---|
|
### Cleanup
sudo virsh destroy edge_k8s-1
sudo virsh undefine edge_k8s-1
sudo virsh pool-destroy edge_k8s-1
sudo virsh pool-undefine edge_k8s-1
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1.qcow2
sudo virsh destroy edge_k8s-2
sudo virsh undefine edge_k8s-2
sudo virsh pool-destroy edge_k8s-2
sudo virsh pool-undefine edge_k8s-2
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2.qcow2 |
Modify sshd_config on VMs
To ensure that user onaplab can successfully ssh into EMCO and edge cluster VMs, add user onaplab to the sshd_config file.
...
| Code Block |
|---|
|
## ssh to each VM:
ssh onaplab@10.121.7.152
sudo -i
cd /etc/ssh
vi sshd_config
AllowUsers ubuntu onaplab
## Save the changes and exit the file |
Deployment Verification
EMCO Deployment Verification
Perform the following steps to verify correct EMCO deployment:
...
To connect to EMCOUI GUI use IP address of amcop-vm-01 and port 30480:
http://10.121.1.52:30480
Edge Cluster Deployment Verification
To verify deployment of Edge Clusters, perform the following steps:
...
| Code Block |
|---|
|
# Determine Edge Cluster VM IP addresses:
[onaplab@os12 ~]$ sudo virsh list --all
Id Name State
----------------------------------------------------
6 amcop-vm-01 running
9 edge_k8s-1 running
10 edge_k8s-2 running
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-1
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet1 52:54:00:19:96:72 ipv4 10.121.7.152/27
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-2
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet2 52:54:00:c0:47:8b ipv4 10.121.7.146/27
# ssh to each VM from the Host Server. You should be able to ssh without specifying the key:
ssh onaplab@10.121.7.152
ssh onaplab@10.121.7.146
# Perform the following tasks inside the VMs:
onaplab@localhost:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cmk-rpgd7 2/2 Running 0 28d
kube-system coredns-dff8fc7d-2xwrk 0/1 Pending 0 28d
kube-system coredns-dff8fc7d-q2gcr 1/1 Running 0 28d
kube-system dns-autoscaler-66498f5c5f-2kzmv 1/1 Running 0 28d
kube-system kube-apiserver-localhost 1/1 Running 0 28d
kube-system kube-controller-manager-localhost 1/1 Running 0 28d
kube-system kube-flannel-8rm9p 1/1 Running 0 28d
kube-system kube-multus-ds-amd64-mt9s5 1/1 Running 0 28d
kube-system kube-proxy-ggk8m 1/1 Running 0 28d
kube-system kube-scheduler-localhost 1/1 Running 0 28d
kube-system kubernetes-dashboard-84999f8b5b-48xjq 1/1 Running 0 28d
kube-system kubernetes-metrics-scraper-54fbb4d595-rw649 1/1 Running 0 28d
kube-system local-volume-provisioner-bmkc6 1/1 Running 0 28d
kube-system virtlet-vk7jl 3/3 Running 0 28d
node-feature-discovery nfd-master-78nms 1/1 Running 0 28d
node-feature-discovery nfd-worker-k4d5g 1/1 Running 45 28d
operator nfn-agent-zlp9g 1/1 Running 0 28d
operator nfn-operator-b768877d8-vcx7v 1/1 Running 0 28d
operator ovn4nfv-cni-4c6rx 1/1 Running 0 28d
# Verify connectivity to EMCO Cluster
onaplab@localhost:~$ ping 10.121.7.145
PING 10.121.7.145 (10.121.7.145) 56(84) bytes of data.
64 bytes from 10.121.7.145: icmp_seq=1 ttl=64 time=0.457 ms
64 bytes from 10.121.7.145: icmp_seq=2 ttl=64 time=0.576 ms
|
Uninstall Guide
Perform the following steps to remove EMCO and Edge Clusters from the Host Server:
...
| Code Block |
|---|
|
sudo virsh destroy amcop-vm-01
sudo virsh undefine amcop-vm-01
sudo virsh pool-destroy amcop-vm-01
sudo virsh pool-undefine amcop-vm-01
sudo rm /var/lib/libvirt/images/amcop-vm-01/amcop-vm-01-cidata.iso
sudo rm /var/lib/libvirt/images/amcop-vm-01/amcop-vm-01.qcow2
sudo virsh destroy edge_k8s-1
sudo virsh undefine edge_k8s-1
sudo virsh pool-destroy edge_k8s-1
sudo virsh pool-undefine edge_k8s-1
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1.qcow2
sudo virsh destroy edge_k8s-2
sudo virsh undefine edge_k8s-2
sudo virsh pool-destroy edge_k8s-2
sudo virsh pool-undefine edge_k8s-2
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2.qcow2
sudo rm -rf ~/amcop_deploy
sudo rm -rf ~/aarna_stream
|
License
References
AMCOP 2.0 Quickstart Guide (Bare Metal) - EMCO Install Guide by Aarna Networks
AMCOP 2.0 User Guide - EMCO Config Guide by Aarna Networks
...