Page History

Introduction

This document describes the blueprint test environment for the Smart Data Transaction for CPS blueprint. The test results and logs are posted in the Akraino Nexus at the link below:Insert link here

https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/

Akarino Test Group Information

...

The test script will start the registry, pull upstream images and populate the registry, clean images left over from the pull process, stop the registry, and remove the registry. The robot command should report success for all test cases.

Test Results

Pass

CI/CD Regression Tests: Node Setup

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/lfedge-docker/1/

Image Added

Pass (5/5 test cases)

CI/CD Regression Tests: Node Setup

This set of test cases confirms This set of test cases confirms the scripting to initialize master and edge nodes.

...

The test scripts will initialize the master and edge nodes and verify the required software is installed. The robot command should report success for all test cases.

Test Results

Pass

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/lfedge-install/1/

Image Added

Pass (2/2 test cases)

CI/CD Regression Tests: Cluster Setup & CI/CD Regression Tests: Cluster Setup & Teardown

These test cases verify that the Kubernetes cluster can be initialized, edge nodes added to it and removed, and the cluster torn down.

...

The test scripts will start the cluster, add all configured edge nodes, remove the edge nodes, and reset the cluster. The robot command should report success for all test cases.

Test Results

Pass

CI/CD Regression Tests: EdgeX Services

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/lfedge-cluster/7/

Image Added

Pass (4/4 test cases)

CI/CD Regression Tests: EdgeX Services

These test cases verify that the EdgeX micro-services can be started and that MQTT messages are passed to the master node from the services.

...

The test scripts will start the EdgeX micro-services on all edge nodes, confirm that MQTT messages are being delivered from the edge nodes, and stop the EdgeX micro-services. The robot command should report success for all test cases.

Test Results

Pass

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/edgex-install/7/

Image Added

Pass (8/8 test cases)

CI/CD Regression Tests: LoRa Device ServiceCI/CD Regression Tests: LoRa Device Service

These test cases verify that the LoRa device service can read sensor data over the LoRa communications channel.

...

The Robot Framework should report success for all test cases.

Test Results

Pass

Feature Project Tests

N/A

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/edgex-lora/3/

Image Added

Pass (2/2 test cases)

Feature Project Tests

N/A

BluVal BluVal Tests

BluVal tests for Lynis, Vuls, and Kube-Hunter were executed on the test bed.

The Test inputs

...

Bluval User Guide

Steps To Implement Security Scan Requirements

Test Procedure

https://vuls.io/docs/en/tutorial-docker.html

Test Procedure

1. Deploy a Test Deploy a Test VM
2. Copy the folder ~/.kube from Kubernetes master node to the Test VM
3. Create SSH Key to access Kubernetes master node

...

We use Ubuntu 20.04, so we run ran Vuls test as follow:Steps To Implement Security Scan Requirements#Vulsfollows:

1. Create directory

   $ mkdir ~/vuls

   
   

   $ cd ~/vuls

   
   

   $ mkdir go-cve-dictionary-log goval-dictionary-log gost-log

   
   

2. Fetch NVD

   $ docker run --rm -it \

   
   

   -v $PWD:/go-cve-dictionary \

   
   

   -v $PWD/go-cve-dictionary-log:/var/log/go-cve-dictionary \

   
   

   vuls/go-cve-dictionary fetch nvd

   
   

3. Fetch OVAL

   $ docker run --rm -it

   \
   

   \ -v $PWD:/goval-dictionary

   \
   

   \ -v $PWD/goval-dictionary-log:/var/log/goval-dictionary

   \
   

   \ vuls/goval-dictionary fetch ubuntu 16 17 18 19 20

   
   

4. Fetch gost

   $ docker run --rm -i

   \
   

   \ -v $PWD:/gost

   \
   -v

   \ -v $PWD/gost-log:/var/log/gost

   \
   

   \ vuls/gost fetch ubuntu

   
   

5. Create config.toml

   [servers]

   [servers.master]

   
   

   host = "192.168.

   2

   51.

   16

   22"

   
   

   port = "22"

   
   

   user = "test-user"

   
   

   keyPath = "/root/.ssh/id_rsa"

    

   # path to ssh private key in docker

   
   

6. Start vuls container to run tests

   $ docker run --rm -it \

   
   

   -v ~/.ssh:/root/.ssh:ro \

   
   

   -v $PWD:/vuls \

   
   

   -v $PWD/vuls-log:/var/log/vuls \

   
   

   -v /etc/localtime:/etc/localtime:ro \

   
   -e "TZ=Asia/Tokyo" \
   

   -v /etc/timezone:/etc/timezone:ro \ vuls/vuls scan \

   
   

   -config=./config.toml

   
   

7. Get the report

   $ docker run --rm -it

   \
   

   \ -v ~/.ssh:/root/.ssh:ro

   \
   

   \ -v $PWD:/vuls

   \
   

   \ -v $PWD/vuls-log:/var/log/vuls

   \
   

   \ -v /etc/localtime:/etc/localtime:ro

   \
   

   \ vuls/vuls report

   \
   

   \ -format-list

   \
   

   \ -config=./config.toml

   
   

Lynis

...

Kuber-Hunter

Expected output

Test Results

Test Dashboards

Single pane view of how the test score looks like for the Blue print.

...

/Kube-Hunter

1. Create ~/validation/bluval/bluval-sdtfc.yaml to customize the Test

   blueprint: name: sdtfc layers: - os - k8s os: &os - name: lynis what: lynis optional: "False" k8s: &k8s - name: kube-hunter what: kube-hunter optional: "False"

   
   

2. Update ~/validation/bluval/volumes.yaml file

   volumes: # location of the ssh key to access the cluster ssh_key_dir: local: '/home/ubuntu/.ssh' target: '/root/.ssh' # location of the k8s access files (config file, certificates, keys) kube_config_dir: local: '/home/ubuntu/kube' target: '/root/.kube/' # location of the customized variables.yaml custom_variables_file: local: '/home/ubuntu/validation/tests/variables.yaml' target: '/opt/akraino/validation/tests/variables.yaml' # location of the bluval-<blueprint>.yaml file blueprint_dir: local: '/home/ubuntu/validation/bluval' target: '/opt/akraino/validation/bluval' # location on where to store the results on the local jumpserver results_dir: local: '/home/ubuntu/results' target: '/opt/akraino/results' # location on where to store openrc file openrc: local: '' target: '/root/openrc' # parameters that will be passed to the container at each layer layers: # volumes mounted at all layers; volumes specific for a different layer are below common: - custom_variables_file - blueprint_dir - results_dir hardware: - ssh_key_dir os: - ssh_key_dir networking: - ssh_key_dir docker: - ssh_key_dir k8s: - ssh_key_dir - kube_config_dir k8s_networking: - ssh_key_dir - kube_config_dir openstack: - openrc sds: sdn: vim:

   
   

3. Update ~/validation/tests/variables.yaml file

   ### Input variables cluster's master host host: <IP Address> # cluster's master host address username: <username> # login name to connect to cluster password: <password> # login password to connect to cluster ssh_keyfile: /root/.ssh/id_rsa # Identity file for authentication

   
   

4. Run Blucon

   $ bash validation/bluval/blucon.sh sdtfc

   
   

Expected output

BluVal tests should report success for all test cases.

Test Results

Vuls results (manual) Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-vuls/2/

Lynis results (manual) Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-lynis/3/

Kube-Hunter results Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-bluval/2/

Vuls

Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-vuls/2/

There are 17 CVEs with a CVSS score >= 9.0. These are exceptions requested here:

Release 6: Akraino CVE and KHV Vulnerability Exception Request

Lynis

Nexus URL (run via Bluval, without fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-bluval/2/

Nexus URL (manual run, with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-lynis/2/

Image Added

The initial results compare with the Lynis Incubation: PASS/FAIL Criteria, v1.0 as follows.

The Lynis Program Update test MUST pass with no errors.

2022-03-04 15:33:28 Test: Checking for program update...
2022-03-04 15:33:31 Current installed version  : 301
2022-03-04 15:33:31 Latest stable version      : 307
2022-03-04 15:33:31 Minimum required version   : 297
2022-03-04 15:33:31 Result: newer Lynis release available!
2022-03-04 15:33:31 Suggestion: Version of Lynis outdated, consider upgrading to the latest version [test:LYNIS] [details:-] [solution:-]

Fix: Download and run the latest Lynis directly on SUT.

Steps To Implement Security Scan Requirements#InstallandExecute

The following list of tests MUST complete as passing

Results after the above fixes are as follows:

The Lynis Program Update test MUST pass with no errors.

2022-03-07 15:19:07 Test: Checking for program update...
2022-03-07 15:19:10 Current installed version : 308
2022-03-07 15:19:10 Latest stable version : 307
2022-03-07 15:19:10 No Lynis update available.

The following list of tests MUST complete as passing

The post-fix manual logs can be found at https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-lynis/3/.

Kube-Hunter

Nexus URL (with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r6/sdt-bluval/2/

Image Added

There are 5 Vulnerabilities.

• KHV002
• KHV005
• KHV050
• CAP_NET_RAW Enabled
• Access to pod's secrets

Fix for KHV002

$ kubectl replace -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "false"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:public-info-viewer
rules:
- nonResourceURLs:
  - /healthz
  - /livez
  - /readyz
  verbs:
  - get
EOF

Fix for KHV005, KHV050, Access to pod's secrets

$ kubectl replace -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
automountServiceAccountToken: false
EOF

The above fixes are implemented in the Ansible playbook deploy/playbook/init_cluster.yml and configuration file deploy/playbook/k8s/fix.yml

Fix for CAP_NET_RAW Enabled:

Create a PodSecurityPolicy with requiredDropCapabilities: NET_RAW. The policy is shown below. The complete fix is implemented in the Ansible playbook deploy/playbook/init_cluster.yml and configuration files deploy/playbook/k8s/default-psp.yml and deploy/playbook/k8s/system-psp.yml, plus enabling PodSecurityPolicy checking in deploy/playbook/k8s/config.yml.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp-baseline
spec:
  privileged: true
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - IPC_LOCK
  - NET_ADMIN
  requiredDropCapabilities:
  - NET_RAW
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - max: 65535
    min: 0
  readOnlyRootFilesystem: false
  fsGroup:
    rule: 'RunAsAny'
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  volumes:
  - '*'

Results after fixes are shown below:

Image Added

Note that in spite of all Kube-Hunter vulnerabilities being fixed, the results still show one test failure. The "Inside-a-Pod Scanning" test case reports failure, apparently because the log ends with "Kube Hunter couldn't find any clusters" instead of "No vulnerabilities were found." Because vulnerabilities were detected and reported earlier by this test case, and those vulnerabilities are no longer reported, we believe this is a false negative, and may be caused by this issue: https://github.com/aquasecurity/kube-hunter/issues/358

Test Dashboards

Single pane view of how the test score looks like for the Blue print.

*Vuls is counted as one test case.

*One Kube-Hunter failure is counted as a pass. See above.

Vuls and Lynis test cases are failing, an exception request is filed for Vuls-detected vulnerabilities that cannot be fixed. The Lynis results have been confirmed to pass the Incubation criteria.

Additional Testing

None at this time.

Bottlenecks/Errata

None at this time.

Additional Testing

...