This wiki documents KubeEdge BP test environment set up and test results for release 4.
All test results and logs have been pushed to Akraino NEXUS:
https://nexus.akraino.org/content/sites/logs/futurewei/kubeedgees/36/
N/A
2 independent test labs have been set up:
They share the same high level test architecture:
In the AWS lab, all servers are EC2 instances:
Node Type | OS |
---|---|
Jump Host | Ubuntu 18.04 |
KubeEdge Cloud | Ubuntu 20.04 |
KubeEdge Edge | Ubuntu 20.04 |
Robot framework is used in BluVal.
N/A
Pass
Pass
We used Ubuntu 20.04. All the packages have been updated or upgraded to latest version in the repo.
There are 4 CVEs with CVSS score > 9.0. These require upstream kernel patches, and exceptions have been requested here:
Akraino CVE Vulnerability Exception Request
Fixed 3 vulnerabilities:
KHV002, KHV050 & CAP_NET_RAW.
No more to fix.
Sonobuoy does not apply to KubeEdge.
An exception has been granted:
Akraino BluVal Exception Request
Reason:
Single pane view of how the test score looks like for the Blue print.
Total Tests | Test Executed | Pass | Fail | In Progress |
---|---|---|---|---|
2 | 2 | 2 | 0 | 0 |
Test results and logs are all pushed to Akraino NEXUS:
https://nexus.akraino.org/content/sites/logs/futurewei/kubeedgees/36/
https://aquasecurity.github.io/kube-hunter/kb/KHV002.html
Solution:
Change the default ClusterRole system:public-info-viewer
kubectl replace -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "false" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:public-info-viewer rules: - nonResourceURLs: - /healthz - /livez - /readyz verbs: - get EOF
Docker runtime enables Linux "NET_RAW" capability by default. Docker daemon does not have an option to disable "NET_RAW":
https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file
So we have to turn to K8s provided "PodSecurityPolicy" for help.
Solution:
Use PodSecurityPolicy
https://kubernetes.io/docs/concepts/policy/pod-security-policy/
High level steps:
Exact PodSecurityPolicy Spec we are using:
spec:
allowPrivilegeEscalation: true
fsGroup:
rule: RunAsAny
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- max: 65535
min: 0
privileged: true
requiredDropCapabilities:
- NET_RAW
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
Solution:
kubectl replace -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: default
automountServiceAccountToken: false
EOF
set -euo pipefail cwd=$(pwd) results_dir=$cwd/results sudo rm -fr $results_dir sudo rm -f $cwd/results.zip mkdir -p $results_dir blueprint=kubeedgees info () { logger -s -t "run_bluval.info" "$*" } error () { logger -s -t "run_bluval.error" "$*" exit 1 } cwd=$cwd/validation cd $cwd # update information in volumes yaml sed -i \ -e "/ssh_key_dir/{n; s@local: ''@local: '$SSH_KEY_DIR'@}" \ -e "/kube_config_dir/{n; s@local: ''@local: '$K8S_CONFIG_DIR'@}" \ -e "/custom_variables_file/{n; s@local: ''@local: '$cwd/tests/variables.yaml'@}" \ -e "/blueprint_dir/{n; s@local: ''@local: '$cwd/bluval/'@}" \ -e "/results_dir/{n; s@local: ''@local: '$results_dir'@}" \ "$cwd/bluval/volumes.yaml" sed -i \ -e "s/host: [0-9]*.[0-9]*.[0-9]*.[0-9]*/host: $CLUSTER_MASTER_IP/" \ -e "s/username: [A-Za-z0-9_]* /username: $SSH_USER/" \ -e "s/password: [A-Za-z0-9_]* /password: /" \ -e "s|ssh_keyfile: [A-Za-z0-9_]* |ssh_keyfile: /root/.ssh/id_rsa|" \ "$cwd/tests/variables.yaml" cat >"$cwd/bluval/bluval-kubeedgees.yaml" <<EOF blueprint: name: kubeedgees layers: - os - k8s os: &os - name: lynis what: lynis optional: "False" - name: vuls what: vuls optional: "False" k8s: &k8s - name: kube-hunter what: kube-hunter optional: "False" EOF $cwd/bluval/blucon.sh $blueprint if [ $? -ne 0 ]; then sudo chown -R $(id -u):$(id -g) "$results_dir" error "blucon.sh exited with return code $?" fi sudo chown -R $(id -u):$(id -g) "$results_dir" echo $BLUEPRINT_BUILD_VERSION source ~/.lftools/bin/activate NEXUS_PATH="${LAB_SILO}/$blueprint/$BLUEPRINT_BUILD_VERSION" cd "$results_dir/.." && zip -r results.zip ./results lftools deploy nexus-zip https://nexus.akraino.org/ logs "$NEXUS_PATH" results.zip |
Check if compilers like gcc, g++ is installed. If yes, remove them. For example on ubuntu:
sudo apt remove gcc g++