...
This document describes steps required to deploy a sample environment for the Public Cloud Edge Interface (PCEI) Blueprint.
License
TBD
Deployment Architecture
The deployment architecture is shown below. All Addressing and naming shown are for example purposes only.
...
| Requirement | Value |
|---|
| CPU | 18 Core / 36 vCPU |
| RAM | 128 GB |
| DISK | 500 GB |
| NETWORK | 1Gbps 1 Gbps NIC Port |
Network Requirements
...
- Install CentOS on Host Server.
- Prepare Host Server for EMCO Deployment.
- Deploy EMCO.
- Deploy Edge Clusters.
Deployment Guide
Install CentOS on Host Server
...
| Code Block |
|---|
|
sudo -i
visudo
# Uncomment the line below
%wheel ALL=(ALL) NOPASSWD: ALL |
Step 2. Add sudo user to wheel group
...
| Code Block |
|---|
|
usermod –aG wheel onaplab |
...
| Code Block |
|---|
|
# Login as super user
sudo -i
# Check for the following parameter
cat /sys/module/kvm_intel/parameters/nested
N
# If it is Y, there is nothing else to be done. It is already enabled!
# If it is N, do the following:
# Edit /etc/default/grub file, and include kvm-intel.nested=1
GRUB_CMDLINE_LINUX parameter
GRUB_CMDLINE_LINUX="crashkernel=auto
console=ttyS0,38400n8,kvm-intel.nested=1"
# Rebuild your GRUB configuration
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# Enable nested KVM capabilities in /etc/modprobe.d/kvm.conf
# By uncommenting the below line
options kvm_intel nested=1
### Reboot the server
reboot
# Now, we should have the nested KVM capabilities enabled
cat /sys/module/kvm_intel/parameters/nested
Y |
Step 4. Install VNC Server (Optional)
Follow instructions at:
https://www.tecmint.com/install-and-configure-vnc-server-in-centos-7/
...
| Code Block |
|---|
|
cat <<\EOF >> netdefault.xml
<network>
<name>default</name>
<bridge name="vmbr0"/>
<forward mode='route' dev='eno24'/>
# <forward/>
<ip address="10.121.7.129" netmask="255.255.255.224">
<dhcp>
<range start="10.121.7.144" end="10.121.7.158"/>
</dhcp>
</ip>
</network>
EOF
sudo virsh net-list
sudo virsh net-destroy default
sudo virsh net-undefine default
sudo virsh net-define netdefault.xml
sudo virsh net-start default
sudo virsh net-autostart default |
...
| Code Block |
|---|
|
# Run commands below on the Host Server
ssh-keygen
cd ~/.ssh
chmod 600 id_rsa
chmod 600 id_rsa.pub
chmod 700 config
chmod 600 known_hosts
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys
echo "# Increase the server timeout value" >> ~/.ssh/config
echo "ServerAliveInterval 120" >> ~/.ssh/config |
Step 2. Download software and install EMCO
...
| Code Block |
|---|
|
#### On the Host Server
sudo yum install -y git deltarpm
mkdir -p amcop_deploy
cd amcop_deploy
## Download the installation package zip file from
wget --load-cookies /tmp/cookies.txt "https://drivedocs.google.com/file/d/1aFtChA454uAMuoEyV4m-jWiRDrZGyYgx/view?usp=sharinguc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1NE6dfbt3l6u2D6MRocdL04_xMjPn06HZ' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1NE6dfbt3l6u2D6MRocdL04_xMjPn06HZ" -O amcop_install_v1.0.zip && rm -rf /tmp/cookies.txt
unzip amcop_install_v1.0.zip
sudo chown -R onaplab:onaplab ~/amcop_deploy/
cd ~/amcop_deploy/aarna-stream/util-scripts
./prep_baremetal_centos.sh
####### Install EMCO/AMCOP
# Edit inventory.ini file. Use IP address of Host Server and the username.
cd ~/amcop_deploy/aarna-stream/anod_lite/ansible/deployment
vi inventory.ini
[deployment_host]
10.121.1.12 ansible_user=onaplab
nohup ansible-playbook ./main.yml -i inventory.ini -e deployment_env=on-prem -e jump_host_user=onaplab --private-key=/home/onaplab/.ssh/id_rsa -e vm_user=onaplab & |
...
| Code Block |
|---|
|
# On the Host Server
cd /home/onaplab/amcop_deploy/aarna-stream/util-scripts
# Add "--cpu host" option to the end of the below line
vi create_qem_vm.sh
virt-install --connect qemu:///system --name $vm_name --ram $(($mem << 10)) --vcpus=$vCPU --os-type linux --os-variant $os_variant --disk path=/var/lib/libvirt/images/$vm_name/"$vm_name".qcow2,format=qcow2 --disk /var/lib/libvirt/images/$vm_name/$vm_name-cidata.iso,device=cdrom --import --network network=default --noautoconsole —cpu—-cpu host
# Save the file |
Step 2. Deploy two Edge Cluster VMs.
...
| Code Block |
|---|
|
# Find VM's IP addresses. On the Host Server run:
[onaplab@os12 ~]$ sudo virsh list --all
Id Name State
----------------------------------------------------
6 amcop-vm-01 running
9 edge_k8s-1 running
10 edge_k8s-2 running
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-1
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet1 52:54:00:19:96:72 ipv4 10.121.7.152/27
[onaplab@os12 ~]$
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-2
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet2 52:54:00:c0:47:8b ipv4 10.121.7.146/27
# ssh to each VM from the Host Server:
ssh onaplab@10.121.7.152
ssh onaplab@10.121.7.146/27
# ssh to Perform the following tasks in each VM from the Host Server:
ssh -i ~/.ssh/id_rsa.pub onaplab@10.121.7.152
ssh -i ~/.ssh/id_rsa.pub onaplab@10.121.7.146
# Perform the following tasks in each VM:
sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y python-pip
git clone https://git.onap.org/multicloud/k8s/
# Run script to setup KUD clusters
nohup k8s/kud/hosting_providers/baremetal/aio.sh %
|
If the edge cluster deployment fails for any reason, please do the clean up steps below before you retry:
:
sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y python-pip
git clone https://git.onap.org/multicloud/k8s/
# Run script to setup KUD clusters
nohup k8s/kud/hosting_providers/baremetal/aio.sh %
|
If the edge cluster deployment fails for any reason, please do the clean up steps below before you retry:
| Code Block |
|---|
|
### Cleanup
sudo virsh destroy edge_k8s-1
sudo virsh undefine edge_k8s-1
sudo virsh pool-destroy edge_k8s-1
sudo virsh pool-undefine edge_k8s-1
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1.qcow2
|
| Code Block |
|---|
|
### Cleanup
sudo virsh destroy edge_k8s-12
sudo virsh undefine edge_k8s-12
sudo virsh pool-destroy edge_k8s-12
sudo virsh pool-undefine edge_k8s-12
sudo rm /var/lib/libvirt/images/edge_k8s-12/edge_k8s-12-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-12/edge_k8s-1.qcow2
sudo virsh destroy edge_k8s-2
sudo virsh undefine edge_k8s-2
sudo virsh pool-destroy edge_k8s-2
sudo virsh pool-undefine edge_k8s-2
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2.qcow22.qcow2 |
Modify sshd_config on VMs
To ensure that user onaplab can successfully ssh into EMCO and edge cluster VMs, add user onaplab to the sshd_config file.
| Code Block |
|---|
|
## ssh to each VM:
ssh onaplab@10.121.7.152
sudo -i
cd /etc/ssh
vi sshd_config
AllowUsers ubuntu onaplab
## Save the changes and exit the file |
Deployment Verification
EMCO Deployment Verification
...
| Code Block |
|---|
|
# Determine IP address of EMCO VM:
[onaplab@os12 ~]$ sudo virsh list --all
Id Name State
----------------------------------------------------
6 amcop-vm-01 running
9 edge_k8s-1 running
10 edge_k8s-2 running
[onaplab@os12 ~]$ sudo domifaddr amcop-vm-01
sudo: domifaddr: command not foundrunning
[onaplab@os12 ~]$ sudo virsh domifaddr amcop-vm-01
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet0 52:54:00:1a:8e:8b ipv4 10.121.7.145/27
# ssh to EMCO VM. You should be able to ssh without specifying the key:
[onaplab@os12 ~]$ ssh onaplab@10.121.7.145
# Verify K8S pods:
onaplab@emco:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6f954885fb-bb2mr 1/1 Running 0 28d
kube-system calico-node-ldcpv 1/1 Running 0 28d
kube-system coredns-6b968665c4-6558h 1/1 Running 0 28d
kube-system coredns-6b968665c4-rq6bf 0/1 Pending 0 28d
kube-system dns-autoscaler-5fc5fdbf6-njch7 1/1 Running 0 28d
kube-system kube-apiserver-node1 1/1 Running 0 28d
kube-system kube-controller-manager-node1 1/1 Running 0 28d
kube-system kube-proxy-gg95d 1/1 Running 0 28d
kube-system kube-scheduler-node1 1/1 Running 0 28d
kube-system kubernetes-dashboard-6c7466966c-cjpxm 1/1 Running 0 28d
kube-system nodelocaldns-7pxcs 1/1 Running 0 28d
kube-system tiller-deploy-8756df4d9-zq52m 1/1 Running 0 28d
onap dev-cassandra-0 1/1 Running 0 28d
onap dev-cassandra-1 1/1 Running 0 28d
onap dev-cassandra-2 1/1 Running 0 28d
onap dev-cds-blueprints-processor-6d697cc4d6-wzlfj 0/1 Init:1/3 0 28d
onap dev-cds-db-0 1/1 Running 0 28d
onap dev-cds-py-executor-7dcdc5f7f6-tpfmg 1/1 Running 0 28d
onap dev-cds-sdc-listener-f99d4588d-nt2tk 0/1 Init:0/1 4021 28d
onap dev-cds-ui-7768bb4b-cfbzd 1/1 Running 0 28d
onap dev-mariadb-galera-0 1/1 Running 0 28d
onap dev-mariadb-galera-1 1/1 Running 0 28d
onap dev-mariadb-galera-2 1/1 Running 0 28d
onap4k8s clm-668c45d96d-99gpb 1/1 Running 0 28d
onap4k8s emcoui-57846bd5df-c774f 1/1 Running 0 28d
onap4k8s etcd-768d5b6cc-ptmmr 1/1 Running 0 28d
onap4k8s middleend-6d67c9bf54-tvs7s 1/1 Running 0 28d
onap4k8s mongo-7988cb488b-kf29q 1/1 Running 0 28d
onap4k8s ncm-9f4b85787-nqnlm 1/1 Running 0 28d
onap4k8s orchestrator-5fd4845f8f-qsxlf 1/1 Running 0 28d
onap4k8s ovnaction-f794f65b6-w85ms 1/1 Running 0 28d
onap4k8s rsync-7d9f5fbd9b-r72sp 1/1 Running 0 28d
# Verify K8S services:
onaplab@emco:~$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 28d
kube-system coredns ClusterIP 10.233.0.3 <none> 53/UDP,53/TCP,9153/TCP 28d
kube-system kubernetes-dashboard ClusterIP 10.233.35.188 <none> 443/TCP 28d
kube-system tiller-deploy ClusterIP 10.233.33.249 <none> 44134/TCP 28d
onap cassandra ClusterIP None <none> 7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP,61621/TCP 28d
onap cds-blueprints-processor-cluster ClusterIP 10.233.4.219 <none> 5701/TCP 28d
onap cds-blueprints-processor-grpc ClusterIP 10.233.34.1 <none> 9111/TCP 28d
onap cds-blueprints-processor-http ClusterIP 10.233.31.74 <none> 8080/TCPTCP 28d
onap cds-db ClusterIP None <none> 3306/TCP 28d
onap cds-db 28d
onap ClusterIPcds-py-executor None <none> ClusterIP 3306/TCP10.233.43.240 <none> 50052/TCP,50053/TCP 28d
onap cds-pysdc-executorlistener ClusterIP 10.233.432.24048 <none> 500528080/TCP,50053/TCP 28d
onap cds-sdc-listenerui NodePort ClusterIP 10.233.255.48 19 <none> 80803000:30497/TCP 28d
onap cdsmariadb-uigalera ClusterIP None NodePort 10.233.55.19 <none> 3000:304973306/TCP 28d
onap 28d
onap4k8s mariadb-galera clm ClusterIP None NodePort <none> 10.233.59.50 <none> 3306/TCP 9061:31856/TCP 28d
onap4k8s clm emcoui NodePort 10.233.592.505 <none> 90619080:3185630480/TCP 28d
onap4k8s etcd emcoui NodePort ClusterIP 10.233.254.580 <none> 9080:304802379/TCP,2380/TCP 28d
onap4k8s etcdmiddleend NodePort ClusterIP 10.233.5411.80 225 <none> 23799891:31289/TCP,2380/TCP 28d
onap4k8s mongo middleend NodePort ClusterIP 10.233.1119.225133 <none> 9891:3128927017/TCP 28d
onap4k8s mongoncm ClusterIPNodePort 10.233.1916.13320 <none> 270179031:32737/TCP 28d
onap4k8s ncm orchestrator NodePort 10.233.1623.2025 <none> 90319015:3273731298/TCP 28d
onap4k8s ovnaction orchestrator NodePort 10.233.2337.2545 <none> 9015:312989053:32514/TCP,9051:31181/TCP 28d
onap4k8s rsync NodePort 28d
onap4k8s 10.233.60.47 ovnaction<none> 9041:30555/TCP NodePort 10.233.37.45 <none> 9053:32514/TCP,9051:31181/TCP 28d
|
Access EMCOUI GUI:
| Code Block |
|---|
|
# Determine EMCOUI Service Port:
onaplab@emco:~$ kubectl get svc emcoui -n onap4k8s
NAME TYPE 28d
onap4k8s CLUSTER-IP rsyncEXTERNAL-IP PORT(S) AGE
emcoui NodePort 10.233.2.5 <none> NodePort9080:30480/TCP 28d
# To connect to EMCOUI GUI use IP address of amcop-vm-01 and port 30480 |
To connect to EMCOUI GUI use IP address of amcop-vm-01 and port 30480:
http://10.121.1.145:30480
Image Added
Edge Cluster Deployment Verification
To verify deployment of Edge Clusters, perform the following steps:
| Code Block |
|---|
|
# Determine Edge Cluster VM IP addresses:
[onaplab@os12 ~]$ sudo virsh list --all
Id Name 10.233.60.47 <none> 9041:30555/TCP 28d
|
Access EMCOUI GUI:
| Code Block |
|---|
|
# Determine EMCOUI Service Port:
onaplab@emco:~$ kubectl get svc emcoui -n onap4k8s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)State
----------------------------------------------------
6 amcop-vm-01 AGE
emcoui NodePort running
9 10.233.2.5 <none> edge_k8s-1 9080:30480/TCP 28d
# To connect to EMCOUI GUI use IP address of amcop-vm-01 and port 30480 |
To connect to EMCOUI GUI use IP address of amcop-vm-01 and port 30480:
http://10.121.1.145:30480
Image Removed
Edge Cluster Deployment Verification
To verify deployment of Edge Clusters, perform the following steps:
| Code Block |
|---|
|
# Determine Edge Cluster VM IP addresses:running
10 edge_k8s-2 running
[onaplab@os12 ~]$ sudo virsh listdomifaddr edge_k8s--all1
Id Name MAC address Protocol State
Address
-----------------------------------------------------------------------------
6 amcop-vm-01 running
9vnet1 edge_k8s-1 52:54:00:19:96:72 ipv4 running
10 edge_k8s-2 running 10.121.7.152/27
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-12
Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet1vnet2 52:54:00:19c0:9647:728b ipv4 10.121.7.146/27
# ssh to each VM from the Host Server. You should be able to ssh without specifying the key:
ssh onaplab@10.121.7.152
ssh onaplab@10.121.7.146
# Perform the following tasks inside the VMs:
onaplab@localhost:~$ kubectl get pods --all-namespaces
NAMESPACE 10.121.7.152/27
[onaplab@os12 ~]$ sudo virsh domifaddr edge_k8s-2
Name MACNAME address Protocol Address
-------------------------------------------------------------------------------
vnet2 52:54:00:c0:47:8b READY ipv4 STATUS RESTARTS 10.121.7.146/27
# ssh to each VM from the Host Server. You should be able to ssh without specifying the key:
ssh onaplab@10.121.7.152
ssh onaplab@10.121.7.146
# Perform the following tasks inside the VMs:
onaplab@localhost:~$ kubectl get pods --all-namespaces
NAMESPACEAGE
kube-system cmk-rpgd7 2/2 Running NAME0 28d
kube-system coredns-dff8fc7d-2xwrk READY STATUS 0/1 RESTARTS AGE
kube-system Pending 0 cmk28d
kube-rpgd7system coredns-dff8fc7d-q2gcr 21/21 Running 0 28d
kube-system corednsdns-autoscaler-dff8fc7d66498f5c5f-2xwrk2kzmv 01/1 PendingRunning 0 28d
kube-system corednskube-dff8fc7dapiserver-q2gcr localhost 1/1 Running 0 28d
kube-system dnskube-autoscalercontroller-66498f5c5fmanager-2kzmvlocalhost 1/1 Running 0 28d
kube-system kube-apiserver-localhostflannel-8rm9p 1/1 Running 0 28d
kube-system kube-multus-controllerds-manager-localhostamd64-mt9s5 1/1 Running 0 28d
kube-system kube-flannelproxy-8rm9pggk8m 1/1 Running 0 28d
kube-system kube-multus-ds-amd64-mt9s5scheduler-localhost 1/1 Running 0 28d
kube-system kubekubernetes-dashboard-proxy-ggk8m84999f8b5b-48xjq 1/1 Running 0 28d
kube-system kubernetes-metrics-scraper-54fbb4d595-rw649 1/1 Running 0 28d
kube-system kube-scheduler-localhost local-volume-provisioner-bmkc6 1/1 Running 0 28d
kube-system kubernetes-dashboard-84999f8b5b-48xjqvirtlet-vk7jl 1/1 Running 0 28d
kube-system kubernetes-metrics-scraper-54fbb4d595-rw649 1/13/3 Running 0 28d
kube-systemnode-feature-discovery nfd-master-78nms local-volume-provisioner-bmkc6 1/1 Running 0 28d
kubenode-feature-systemdiscovery nfd-worker-k4d5g virtlet-vk7jl 1/1 Running 45 3/3 Running28d
operator 0 28d
node-feature-discovery nfdnfn-masteragent-78nmszlp9g 1/1 Running 0 28d
node-feature-discoveryoperator nfd-worker-k4d5g nfn-operator-b768877d8-vcx7v 1/1 Running 0 45 28d
operator nfnovn4nfv-agentcni-zlp9g 4c6rx 1/1 Running 0 28d
operator# Verify connectivity to EMCO Cluster
onaplab@localhost:~$ ping 10.121.7.145
PING nfn-operator-b768877d8-vcx7v 1/1 Running 0 28d
operator ovn4nfv-cni-4c6rx 1/1 Running 0 28d
# Verify connectivity to EMCO Cluster
onaplab@localhost:~$ ping 10.121.7.145
PING 10.121.7.145 (10.121.7.145) 56(84) bytes of data.
64 bytes from 10.121.7.145: icmp_seq=1 ttl=64 time=0.457 ms
64 bytes from 10.121.7.145: icmp_seq=2 ttl=64 time=0.576 ms
|
Uninstall Guide
License
...
10.121.7.145 (10.121.7.145) 56(84) bytes of data.
64 bytes from 10.121.7.145: icmp_seq=1 ttl=64 time=0.457 ms
64 bytes from 10.121.7.145: icmp_seq=2 ttl=64 time=0.576 ms
|
Uninstall Guide
Perform the following steps to remove EMCO and Edge Clusters from the Host Server:
| Code Block |
|---|
|
sudo virsh destroy amcop-vm-01
sudo virsh undefine amcop-vm-01
sudo virsh pool-destroy amcop-vm-01
sudo virsh pool-undefine amcop-vm-01
sudo rm /var/lib/libvirt/images/amcop-vm-01/amcop-vm-01-cidata.iso
sudo rm /var/lib/libvirt/images/amcop-vm-01/amcop-vm-01.qcow2
sudo virsh destroy edge_k8s-1
sudo virsh undefine edge_k8s-1
sudo virsh pool-destroy edge_k8s-1
sudo virsh pool-undefine edge_k8s-1
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-1/edge_k8s-1.qcow2
sudo virsh destroy edge_k8s-2
sudo virsh undefine edge_k8s-2
sudo virsh pool-destroy edge_k8s-2
sudo virsh pool-undefine edge_k8s-2
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2-cidata.iso
sudo rm /var/lib/libvirt/images/edge_k8s-2/edge_k8s-2.qcow2
sudo rm -rf ~/amcop_deploy
sudo rm -rf ~/aarna_stream
|
License
References
AMCOP Quickstart Guide (Bare Metal) - EMCO Install Guide by Aarna Networks
AMCOP User Guide - EMCO Config Guide by Aarna Networks