...
This document describes the blueprint test environment for the Smart Data Transaction for CPS blueprint. The test results and logs are posted in the Akraino Nexus at the link below:
https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7
Akarino Test Group Information
...
Tests are carried out on the architecture shown in the diagram below.
Test Bed
The test bed consists of a VM 4 VMs running on x86 hardware, performing jump host/deploy deploy and ci/cd and build and master node roles, two edge nodes on ARM64 (Jetson Nano) hardware, and two sensor nodes on ARM32 (Raspberry Pi) hardware.
Node Type | Count | Hardware | OS | ||
---|---|---|---|---|---|
CI/CD | 1 | Intel i5, 2 cores VM | Ubuntu 20.04 | ||
Build | 1 | Intel i5, 2 cores VM | Ubuntu 20.04 | ||
Deploy | 1 | Intel i5, 2 cores VM | Ubuntu 20.04 | ||
Master | 1 | Intel i5, 2 cores VM | Ubuntu 20.04 | ||
Edge | 2 | Jetson Nano, ARM Cortex-A57, 4 cores | Ubuntu 20.04 | ||
SensorCamera | 2 | Raspberry Pi 3, ARM Cortex-A53, 4 cores | Rasbian 11.1 | H.View HV-500E6A | N/A (pre-installed) |
The Build A second VM is used to run the BluVal test framework components outside the system under test.
...
Traffic Generator
N/A
Test API description
Before running the tests below, ensure that the configuration in the chapter Verifying the Setup
of Smart Data Transaction for CPS R7 Installation Guide has been implemented.
CI/CD Regression Tests:
...
Node Setup
This set of test cases confirms the Docker private registry setup, population, and tear-down proceduresthe scripting to change the default runtime of edge nodes.
The Test inputs
The test scripts and data are stored in the source repository's cicd/tests/sdt_step2/install/docker
directory.
Test Procedure
The test bed is placed place in a state with the deploy and master node setup complete, but with Kubernetes, EdgeX, and the private registry not running.where all nodes are prepared with required software. No EdgeX or Kubernetes services are running.
Execute the test scripts:
robot cicd/tests/dockersdt_step2/install/
Expected output
The test script scripts will change the default runtime of edge nodes from runc to nvidia.
start the registry, pull upstream images and populate the registry, clean images left over from the pull process, stop the registry, and remove the registry. The robot command should report success for all test cases.
Test Results
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lfedge-dockerinstall/114/
Pass (51/5 1 test casescase)
CI/CD Regression Tests:
...
Images Build & Push
These This set of test cases confirms the scripting to initialize master and edge nodesverify that the images for EdgeX microservices can be constructed, and pushed to private registry.
The Test inputs
The test scripts and data are stored in the source repository's cicd/tests/install
directorysdt_step2/build/
directory.
Test Procedure
The test bed is place placed in a state where only the deploy node is initialized. No EdgeX or Kubernetes services are running. For a complete test, the master and edge nodes should not have any software installed that was not installed as part of the OS installation.all nodes are prepared with required software and the Docker registry is running.
Execute the test scripts:
robot cicd/tests/installsdt_step2/build/
Expected output
The test scripts will build images of changed services(sync-app/image-app/device-camera), add push the images to private registry.
initialize the master and edge nodes and verify the required software is installed. The robot command should report success for all test cases.
Test Results
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lfedge-installbuild/1/5
Pass (2/2 test cases)
...
The test scripts and data are stored in the source repository's cicd/tests/sdt_step2/cluster/
directory.
Test Procedure
...
Execute the test scripts:
robot cicd/tests/sdt_step2/cluster/
Expected output
The test scripts will start the cluster, add all configured edge nodes, remove the edge nodes, and reset the cluster.
The robot command should report success for all test cases.
Test Results
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lfedge-cluster/7/6
Pass (4/4 test cases)
...
The test scripts and data are stored in the source repository's cicd/tests/sdt_step2/edgex/
directory.
Test Procedure
The test bed is placed in a state where the cluster is initialized and all edge nodes have joined. The Docker registry and mosquitto MQTT broker must be running on the master node. The registry must be populated with all upstream images and custom images. Either the device-lora
service camera
service should be enabled with dht2lra
service running on the sensor nodes, or device-virtual
should be enabled to provide readings.
Execute the test scripts:
robot cicd/tests/sdt_step2/edgex/
Expected output
The test scripts will start the EdgeX micro-services on all edge nodes, confirm that MQTT messages are being delivered from the edge nodes, and stop the EdgeX micro-services.
The robot command should report success for all test cases.
Test Results
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/edgex-install/7/
Pass (8/8 test cases)
CI/CD Regression Tests:
...
Camera Device Service
These test cases verify that the LoRa device service can read sensor data over the LoRa communications channelthe device-camera
service can get image from IP Camera, the sync-app
service can share the image to other edge node, the image-app
service can analyze the image, and the support-notification can receive the crowded notification.
The Test inputs
The test steps and data are contained in the scripts in the source repository cicd/tests/lorasdt_step2/camera/
directory.
Test Procedure
The test bed is initialized to the point of having all EdgeX services running, with device-lora
enabled.
The dht2lra
service is started on the two sensor nodes.
camera and image-app
enabled.
Execute the test scripts:
robot cicd/tests/lorasdt_step2/camera/
Expected output
The test cases will check if MQTT messages containing temperature data gathered from the sensor nodes are arriving at the master node on the topic for each each edge node, validating that the LoRa device support is functioningand the core-data service containing the data of image acquisition, image sharing and image analysis, and check whether the support-notification service having the notification data of crowded after setting the crowded rule.
The Robot Framework should report success for all test cases.
Test Results
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/edgex-lora/3/sdt/r7/camera/10
Pass (29/2 9 test cases)
Feature Project Tests
...
https://vuls.io/docs/en/tutorial-docker.html
Test Procedure
...
- Copy the folder ~/.kube from Kubernetes master node to the Test Build VM
- Create SSH Key on Build VM to access Kubernetes master node
Vuls
We use Ubuntu 20.04, and behind a proxy, so we ran run Vuls test as follows:
Create directory
$ mkdir ~/vuls $ cd ~/vuls $ mkdir go-cve-dictionary-log goval-dictionary-log gost-log
Fetch NVD
$ docker run --rm -it \ -v $PWD:/go-cve-dictionary \ -v $PWD/go-cve-dictionary-log:/var/log/go-cve-dictionary \ vuls/go-cve-dictionary fetch nvd --http-proxy $http_proxy
Fetch OVAL
$ docker run --rm -it \ -v $PWD:/goval-dictionary \ -v $PWD/goval-dictionary-log:/var/log/goval-dictionary \ vuls/goval-dictionary fetch ubuntu 14 16 17 18 19 20 --http-proxy $http_proxy
Fetch gost
$ docker run --rm -iit \
-v $PWD:/gost \ e http_proxy=$http_proxy \
-v $PWD/gost-log:/e https_proxy=$https_proxy \ -v $PWD:/gost \ -v $PWD/gost-log:/var/log/gost \ vuls/gost fetch ubuntu --http-proxy $http_proxyCreate config.toml
[servers] [servers.master] host = "192.168.51.22" port = "22" user = "test-user" keyPath = "/root/.ssh/id_rsa" # path to ssh private key in docker
Start vuls container to run tests
$ docker run --rm -it \ -v ~/.ssh:/root/.ssh:ro \ -v $PWD:/vuls \ -v $PWD/vuls-log:/var/log/vuls \ -v /etc/localtime:/etc/localtime:ro \ -v /etc/timezone:/etc/timezone:ro \ vuls/vuls scan \ -config=./config.toml \
--http-proxy $http_proxyGet the report
$ docker run --rm -it \ -v ~/.ssh:/root/.ssh:ro \ -v $PWD:/vuls \ -v $PWD/vuls-log:/var/log/vuls \ -v /etc/localtime:/etc/localtime:ro \ vuls/vuls report \ -format-list \ -config=./config.toml
Lynis/Kube-Hunter
\
--http-proxy $http_proxy
Lynis/Kube-Hunter
Create ~/validation/bluval/bluval-sdtfc.yaml to customize the Test
blueprint: name: sdtfc layers: - os k8s
- k8sos osk8s: &osk8s - name: lyniskube-hunter what: lyniskube-hunter optional: "False"
k8sos: &k8s os
-
name: kube-hunter lynis
what: kube-hunter lynis
optional: "False"Update ~/validation/bluval/volumes.yaml file
volumes: # location of the ssh key to access the cluster ssh_key_dir: local: '/home/ubuntu/.ssh' target: '/root/.ssh' # location of the k8s access files (config file, certificates, keys) kube_config_dir: local: '/home/ubuntu/kube' target: '/root/.kube/' # location of the customized variables.yaml custom_variables_file: local: '/home/ubuntu/validation/tests/variables.yaml' target: '/opt/akraino/validation/tests/variables.yaml' # location of the bluval-<blueprint>.yaml file blueprint_dir: local: '/home/ubuntu/validation/bluval' target: '/opt/akraino/validation/bluval' # location on where to store the results on the local jumpserver results_dir: local: '/home/ubuntu/results' target: '/opt/akraino/results' # location on where to store openrc file openrc: local: '' target: '/root/openrc' # parameters that will be passed to the container at each layer layers: # volumes mounted at all layers; volumes specific for a different layer are below common: - custom_variables_file - blueprint_dir - results_dir hardware: - ssh_key_dir os: - ssh_key_dir networking: - ssh_key_dir docker: - ssh_key_dir k8s: - ssh_key_dir - kube_config_dir k8s_networking: - ssh_key_dir - kube_config_dir openstack: - openrc sds: sdn: vim:
Update ~/validation/tests/variables.yaml file
### Input variables cluster's master host host: <IP Address> # cluster's master host address username: <username> # login name to connect to cluster password: <password> # login password to connect to cluster ssh_keyfile: /root/.ssh/id_rsa # Identity file for authentication
Run Blucon
$ bash validation/bluval/blucon.sh sdtfc
...
Vuls results (manual) Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-vuls/2/
Lynis results (manual) Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-lynis/32/
Kube-Hunter results Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-bluval/21/
Vuls
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-vuls//r7/sdt-vuls/2/
There are 17 CVEs with 4 CVEs with a CVSS score >= 9.0. These These are exceptions requested here:
Release 57: Akraino CVE and KHV Vulnerability Exception Request
CVE-ID | CVSS | NVD | Fix/Notes |
CVE- |
2022- |
3643 |
10. |
0 | https://nvd.nist.gov/vuln/detail/CVE- |
2022- |
3643 | Fix not yet available |
CVE- |
2016- |
1585 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE- |
2016- |
1585 | No fix available |
CVE- |
2022- |
0318 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE- |
2022- |
0318 | Fix not yet available |
CVE- |
2022- |
3649 | 9.8 | https://nvd.nist.gov/vuln/detail/CVE- |
2022- |
3649 | Fix not yet available |
Lynis
Nexus URL (manual run, with fixes): https://
...
...
Lynis
Nexus URL (run via Bluval, without fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-bluval/2/
Nexus URL (manual run, with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-lynis/2/
The initial results compare with the Lynis Incubation: PASS/FAIL Criteria, v1.0 as follows.
The Lynis Program Update test MUST pass with no errors.
2022-03-04 15:33:28 Test: Checking for program update...
2022-03-04 15:33:31 Current installed version : 301
2022-03-04 15:33:31 Latest stable version : 307
2022-03-04 15:33:31 Minimum required version : 297
2022-03-04 15:33:31 Result: newer Lynis release available!
2022-03-04 15:33:31 Suggestion: Version of Lynis outdated, consider upgrading to the latest version [test:LYNIS] [details:-] [solution:-]
Fix: Download and run the latest Lynis directly on SUT.
Steps To Implement Security Scan Requirements#InstallandExecute
The following list of tests MUST complete as passing
...
Result: password minimum age is not configured
Suggestion: Configure minimum password age in /etc/login.defs [test:AUTH-9286]
...
Result: found umask 022, which could be improved
Suggestion: Default umask in /etc/login.defs could be more strict like 027 [test:AUTH-9328]
...
Result: SSH has no specific user or group limitation. Most likely all valid users can SSH to this machine.
Hardening: assigned partial number of hardening points (0 of 1).
...
Test: checking for file /etc/network/if-up.d/ntpdate
Result: file /etc/network/if-up.d/ntpdate does not exist
...
Hardening: assigned maximum number of hardening points for this item (3).
...
Set recommended value in /etc/sysctl.d/90-lynis-hardening.conf and disable apport in /etc/default/apport
...
Result: sysctl key net.ipv4.conf.default.accept_source_route has a different value than expected in scan profile. Expected=0, Real=1
...
Result: found installed compiler. See top of logfile which compilers have been found or use /usr/bin/grep to filter on 'compiler'
Hardening: assigned partial number of hardening points (1 of 3).
...
Results after the above fixes are as follows:
The Lynis Program Update test MUST pass with no errors.
2022-03-07 15:19:07 Test: Checking for program update...
2022-03-07 15:19:10 Current installed version : 308
2022-03-07 15:19:10 Latest stable version : 307
2022-03-07 15:19:10 No Lynis update available.
The following list of tests MUST complete as passing
...
Result: max password age is 180 days
Hardening: assigned maximum number of hardening points for this item (3).
...
Result: umask is 027, which is fine
Hardening: assigned maximum number of hardening points for this item (2).
...
Result: SSH is limited to a specific set of users, which is good
Hardening: assigned maximum number of hardening points for this item (2).
...
Result: sysctl key net.ipv4.conf.default.accept_source_route contains equal expected and current value (0)
Hardening: assigned maximum number of hardening points for this item (1).
...
Result: no compilers found
Hardening: assigned maximum number of hardening points for this item (3).
The post-fix manual logs can be found at https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-lynis/3/.
Kube-Hunter
Nexus URL (initial run without fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-bluval/1/
Nexus URL (with fixes): https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt-bluval/2/
There are 5 Vulnerabilities.
- KHV002
- KHV005
- KHV050
- CAP_NET_RAW Enabled
- Access to pod's secrets
Fix for KHV002
...
$ kubectl replace -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "false"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:public-info-viewer
rules:
- nonResourceURLs:
- /healthz
- /livez
- /readyz
verbs:
- get
EOF
akraino.org/content/sites/logs/fujitsu/job/sdt/r7/lynis/3/
The results compare with the Lynis Incubation: PASS/FAIL Criteria, v1.0 as follows.
The Lynis Program Update test MUST pass with no errors.
2022-09-14 16:19:49 Test: Checking for program update...
2022-09-14 16:19:49 Result: Update check failed. No network connection?
2022-09-14 16:19:49 Info: to perform an automatic update check, outbound DNS connections should be allowed (TXT record).
2022-09-14 16:19:49 Suggestion: This release is more than 4 months old. Check the website or GitHub to see if there is an update available. [test:LYNIS] [details:-] [solution:-]
The test environment is a proxied private network inside the Fujitsu corporate network which does not allow direct DNS lookups using tools such as dig. Therefore the update check cannot be performed automatically.
The latest version of Lynis, 3.0.8 at time of execution, was downloaded and run directly on the SUT. See the link below:
Steps To Implement Security Scan Requirements#InstallandExecute
The following list of tests MUST complete as passing
No. | Test | Result | Notes |
---|---|---|---|
1 | Test: Checking PASS_MAX_DAYS option in /etc/login.defs | 2022-12-16 18:45:05 Test: Checking PASS_MAX_DAYS option in /etc/login.defs | Required configuration |
2 | Performing test ID AUTH-9328 (Default umask values) | 2022-12-16 18:45:05 Performing test ID AUTH-9328 (Default umask values) 2022-12-16 18:45:05 Test: Checking /etc/login.defs | Required configuration |
3 | Performing test ID SSH-7440 (Check OpenSSH option: AllowUsers and AllowGroups) | 2022-12-16 18:45:14 Performing test ID SSH-7440 (Check OpenSSH option: AllowUsers and AllowGroups) | Required configuration |
4 | Test: checking for file /etc/network/if-up.d/ntpdate | 2022-12-16 18:45:16 Test: checking for file /etc/network/if-up.d/ntpdate 2022-12-16 18:45:16 Result: file /etc/network/if-up.d/ntpdate does not exist 2022-12-16 18:45:16 Result: Found a time syncing daemon/client. 2022-12-16 18:45:16 Hardening: assigned maximum number of hardening points for this item (3). Currently having 173 points (out of 246) | |
5 | Performing test ID KRNL-6000 (Check sysctl key pairs in scan profile) : Following sub-tests required | N/A | |
5a | sysctl key fs.suid_dumpable contains equal expected and current value (0) | 2022-12-16 18:45:27 Result: sysctl key fs.suid_dumpable contains equal expected and current value (0) | Required configuration |
5b | sysctl key kernel.dmesg_restrict contains equal expected and current value (1) | 2022-12-16 18:45:27 Result: sysctl key kernel.dmesg_restrict contains equal expected and current value (1) | Required configuration |
5c | sysctl key net.ipv4.conf.default.accept_source_route contains equal expected and current value (0) | 2022-12-16 18:45:27 Result: sysctl key net.ipv4.conf.default.accept_source_route contains equal expected and current value (0) | Required configuration |
6 | Test: Check if one or more compilers can be found on the system | 2022-12-16 18:45:28 Performing test ID HRDN-7220 (Check if one or more compilers are installed) | Required removal of build-essential package and apt autoremove, and /bin/as |
Kube-Hunter
Nexus URL: https://nexus.akraino.org/content/sites/logs/fujitsu/job/sdt/r7/sdt-bluval/1/
There are no reported vulnerabilities. Note, this release includes fixes for vulnerabilities found in release 6. See the release 6 test document for details on those vulnerabilities and the fixes.
Note that
Fix for KHV005, KHV050, Access to pod's secrets
...
$ kubectl replace -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: default
automountServiceAccountToken: false
EOF
The above fixes are implemented in the Ansible playbook deploy/playbook/init_cluster.yml
and configuration file deploy/playbook/k8s/fix.yml
Fix for CAP_NET_RAW Enabled:
Create a PodSecurityPolicy with requiredDropCapabilities: NET_RAW. The policy is shown below. The complete fix is implemented in the Ansible playbook deploy/playbook/init_cluster.yml
and configuration files deploy/playbook/k8s/default-psp.yml
and deploy/playbook/k8s/system-psp.yml
, plus enabling PodSecurityPolicy checking in deploy/playbook/k8s/config.yml
.
...
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp-baseline
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- IPC_LOCK
- NET_ADMIN
requiredDropCapabilities:
- NET_RAW
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- max: 65535
min: 0
readOnlyRootFilesystem: false
fsGroup:
rule: 'RunAsAny'
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
volumes:
- '*'
Results after fixes are shown below:
Note that in spite of all Kube-Hunter vulnerabilities being fixed, the results still show one test failure. The "Inside-a-Pod Scanning" test case reports failure, apparently because the log ends with "Kube Hunter couldn't find any clusters" instead of "No vulnerabilities were found." This also occurred during release 6 testing. Because vulnerabilities were detected and reported earlier in release 6 by this test case, and those vulnerabilities are no longer reported, we believe this is a false negative, and may be caused by this issue: https://github.com/aquasecurity/kube-hunter/issues/358
...
Total Tests | Test Executed | Pass | Fail | In Progress |
---|---|---|---|---|
262926 | 29 | 2427 | 2 | 0 |
*Vuls is counted as one test case.
...