Test Network Function
This repository contains a set of Cloud-Native Network Functions (CNFs) test cases and the framework to build more. The tests and framework are intended to test the interaction of CNFs with OpenShift Container Platform. It also generates a report (claim.json) after running tests.
Please consult CATALOG.md for a catalog of the included test cases and test case building blocks.
The suite is provided here in part so that CNF Developers can use the suite to test their CNFs readiness for certification. Please see "CNF Developers" below for more information.
Overview
In the diagram above:
- the
CNF under test
is the CNF to be certified. The certification suite identifies the resources (containers/pods/operators etc) belonging to the CNF via labels or static data entries in the config file - the
Certification container/exec
is the certification test suite running on the platform or in a container. The executable verifies the CNF under test configuration and its interactions with openshift - the
Partner pod
can be any pod with the required tools in the same namespace as theCNF under test
. For example, during connectivity tests, the partner pod will generate pings towards theCNF under test
to verify connectivity. The partner pods/containers are auto deployed by the test suite prior a test run and can be auto discovered by the suite without any data entry in the config file.
Test Configuration
The Test Network Function support auto-configuration using labels and annotations, but also a static configuration using a file. The following sections describe how to configure the TNF via labels/annotation and the corresponding settings in the config file. A sample config file can be found here.
targetPodLabels
The goal of this section is to specify the label to be used to identify the cnf under test pods. So for example, with the default configuration:
targetPodLabels:
- namespace: test-network-function.com
name: generic
value: target
The corresponding label prefix is:
test-network-function.com/generic: target
When labelling a pod to be discovered and tested, the discoered pods are in addition to the ones explicitly configured in the testTarget sections of the config file.
testTarget
podsUnderTest / containersUnderTest
This section is usually not required if labels defined in the section above cover all resources that should be tested. It's highly recommended that the labels shoud be defined in pod definition rather than added after pod is created, as labels added later on will be lost in case the pod gets rescheduled. In case of pods defined as part a deployment, it's best to use the same label as the one defined in the spec.selector.matchLabels
section of the deployment yaml.
The autodiscovery mechanism will also attempt to identify the default network device and all the IP addresses of the pods it needs for network connectivity tests, though that information can be explicitly set using annotations if needed. For Pod IPs:
- The annotation
test-network-function.com/multusips
is the highest priority, and must contain a JSON-encoded list of IP addresses to be tested for the pod. This must be explicitly set. - If the above is not present, the
k8s.v1.cni.cncf.io/networks-status
annotation is checked and all IPs from it are used. This annotation is automatically managed in OpenShift but may not be present in K8s. - If neither of the above is present, then only known IPs associated with the pod are used (the pod
.status.ips
field).
For Network Interfaces:
- The annotation
test-network-function.com/defaultnetworkinterface
is the highest priority, and must contain a JSON-encoded string of the primary network interface for the pod. This must be explicitly set if needed. Examples can be seen in cnf-certification-test-partner - If the above is not present, the
k8s.v1.cni.cncf.io/networks-status
annotation is checked and the"interface"
from the first entry found with"default"=true
is used. This annotation is automatically managed in OpenShift but may not be present in K8s.
If a pod is not suitable for network connectivity tests because it lacks binaries (e.g. ping
), it should be
given the label test-network-function.com/skip_connectivity_tests
to exclude it from those tests. The label value is
not important, only its presence. Equivalent to excludeContainersFromConnectivityTests
in the config file.
If label based discovery is not sufficient, this section can be manually populated as shown in the commented part of the sample config. However, instrusive tests need to be skipped (see here) for a reliable test result.
operators
The section can be configured as well as auto discovered. For manual configuration, see the commented part of the sample config. For auto discovery:
- CSVs to be tested by the
operator
spec are identified with thetest-network-function-com/operator=target
label. Any value is permitted buttarget
is used here for consistency with the other specs. - Defining which tests are to be run on the operator is done using the
test-network-function.com/operator_tests
annotation. This is equivalent to thetest-network-function.com/container_tests
and behaves the same. -
test-network-function.com/subscription_name
is optional and should contain a JSON-encoded string that's the name of the subscription for this CSV. If unset, the CSV name will be used.
testPartner
This section can also be discovered automatically and should be left commented out unless the parter pods are modified from the original version in cnf-certification-test-partner
certifiedcontainerinfo and certifiedoperatorinfo
The certifiedcontainerinfo
and certifiedoperatorinfo
sections contain information about CNFs and Operators that are
to be checked for certification status on Red Hat catalogs.
Runtime environement variables to skip or include tests
Turn off openshift required tests
When test on CNFs that run on k8s only environment, execute shell command below before compile tool and run test shell script.
export TNF_MINIKUBE_ONLY=true
Disable intrusive tests
If you would like to skip intrusive tests which may disrupt cluster operations, issue the following:
export TNF_NON_INTRUSIVE_ONLY=true
Likewise, to enable intrusive tests, set the following:
export TNF_NON_INTRUSIVE_ONLY=false
Execute test suites from openshift-kni/cnf-feature-deploy
The test suites from openshift-kni/cnf-feature-deploy can be run prior to the actual CNF certification test execution and the results are incorporated in the same claim file if the following environment variable is set:
export VERIFY_CNF_FEATURES=true
Currently, these suites are skipped:
- performance
- sriov
- ptp
- sctp
- xt_u32
- dpdk
- ovn
For more information on the test suites, refer to the cnf-features-deploy repository
Running the tests with in a prebuild container
A ready to run container is available at this repository: quay.io
To pull the latest container and run the tests you use the following command. There are several required arguments:
-
-t
gives the local directory that contains tnf config files set up for the test. -
-o
gives the local directory that the test results will be available in once the container exits. - Finally, list the specs to be run must be specified, space-separated.
Optional arguments are:
-
-i
gives a name to a custom TNF container image. Supports local images, as well as images from external registries. -
-k
gives a path to one or more kubeconfig files soto be used by the container to authenticate with the cluster. Paths must be separated by a colon. -
-n
gives the network mode of the container. Defaults tobridge
. See the docker run --network parameter reference for more information on how to configure network settings.
If -k
is not specified, autodiscovery is performed.
The autodiscovery first looks for paths in the $KUBECONFIG
environment variable on the host system, and if the variable is not set or is empty, the default configuration stored in $HOME/.kube/config
is checked.
./run-tnf-container.sh -k ~/.kube/config -t ~/tnf/config -o ~/tnf/output diagnostic access-control
Note: Tests must be specified after all other arguments! see General tests for a list of available keywords.
Note: The run-tnf-container.sh
script performs autodiscovery of selected TNF environment variables.
Currently supported environment variables include:
TNF_MINIKUBE_ONLY
docker
instead of podman
Running using By default, run-container.sh
utilizes podman
. However, you can configure an alternate container virtualization
client using TNF_CONTAINER_CLIENT
. This is particularly useful for operating systems that do not readily support
podman
, such as macOS. In order to configure the test harness to use docker
, issue the following prior to
run-tnf-container.sh
:
export TNF_CONTAINER_CLIENT="docker"
Building the container image locally
You can build an image locally by using the command below. Use the value of TNF_VERSION
to set a branch, a tag, or a hash of a commit that will be installed into the image.
docker build -t test-network-function:v1.0.5 --build-arg TNF_VERSION=v1.0.5 .
To build an image that installs TNF from an unofficial source (e.g. a fork of the TNF repository), use the TNF_SRC_URL
build argument to override the URL to a source repository.
docker build -t test-network-function:v1.0.5 \
--build-arg TNF_VERSION=v1.0.5 \
--build-arg TNF_SRC_URL=https://github.com/test-network-function/test-network-function .
To make run-tnf-container.sh
use the newly built image, specify the custom TNF image using the -i
parameter.
./run-tnf-container.sh -i test-network-function:v1.0.5 -t ~/tnf/config -o ~/tnf/output diagnostic access-control
Note: see General tests for a list of available keywords.
Building and running the standalone test executable
Currently, all available tests are part of the "CNF Certification Test Suite" test suite, which serves as the entrypoint
to run all test specs. CNF Certification 3.0
is not containerized, and involves pulling, building, then running the
tests.
By default, test-network-function
emits results to test-network-function/cnf-certification-tests_junit.xml
.
The included default configuration is for running generic
and multus
suites on the trivial example at
cnf-certification-test-partner. To configure for your
own environment, please see config.md.
Dependencies
At a minimum, the following dependencies must be installed prior to running make install-tools
.
Dependency | Minimum Version |
---|---|
GoLang | 1.14 |
golangci-lint | 1.32.2 |
jq | 1.6 |
OpenShift Client | 4.4 |
Other binary dependencies required to run tests can be installed using the following command:
make install-tools
Finally the source dependencies can be installed with
make update-deps
Note: You must also make sure that $GOBIN
(default $GOPATH/bin
) is on your $PATH
.
Note: Efforts to containerize this offering are considered a work in progress.
Pulling The Code
In order to pull the code, issue the following command:
mkdir ~/workspace
cd ~/workspace
git clone git@github.com:test-network-function/test-network-function.git
cd test-network-function
Building the Tests
In order to build the test executable, first make sure you have satisfied the dependencies.
make build-cnf-tests
Gotcha: The make build*
commands run unit tests where appropriate. They do NOT test the CNF.
Testing a CNF
Once the executable is built, a CNF can be tested by specifying which suites to run using the run-cnf-suites.sh
helper
script.
Run any combination of the suites keywords listed at in the General tests section, e.g.
./run-cnf-suites.sh diagnostic
./run-cnf-suites.sh diagnostic lifecycle
./run-cnf-suites.sh diagnostic networking operator
./run-cnf-suites.sh diagnostic platform-alteration
./run-cnf-suites.sh diagnostic generic lifecycle affiliated-certification operator
By default the claim file will be output into the same location as the test executable. The -o
argument for
run-cnf-suites.sh
can be used to provide a new location that the output files will be saved to. For more detailed
control over the outputs, see the output of test-network-function.test --help
.
cd test-network-function && ./test-network-function.test --help
Gotcha: The generic test suite requires that the CNF has both ping
and ip
binaries installed. Please add them
manually if the CNF under test does not include these. Automated installation of missing dependencies is targeted
for a future version.
Available Test Specs
There are two categories for CNF tests; 'General' and 'CNF-specific' (TODO).
The 'General' tests are designed to test any commodity CNF running on OpenShift, and include specifications such as 'Default' network connectivity.
'CNF-specific' tests are designed to test some unique aspects of the CNF under test are behaving correctly. This could
include specifications such as issuing a GET
request to a web server, or passing traffic through an IPSEC tunnel.
'CNF-specific' test are yet to be defined.
General tests
Test in the "general" category belong to multiple suites that can be run in any combination as is appropriate for the CNF(s) under test. Test suites group tests by topic area:
Suite | Test Spec Description | Minimum OpenShift Version |
---|---|---|
access-control |
The access-control test suite is used to test service account, namespace and cluster/pod role binding for the pods under test. It also tests the pods/containers configuration. | 4.4.3 |
affiliated-certification |
The affiliated-certification test suite verifies that the containers in the pod under test and operator under test are certified by Redhat | 4.4.3 |
diagnostic |
The diagnostic test suite is used to gather node information from an OpenShift cluster. The diagnostic test suite should be run whenever generating a claim.json file. | 4.4.3 |
lifecycle |
The lifecycle test suite verifies the pods deployment, creation, shutdown and survivability. | 4.4.3 |
networking |
The networking test suite contains tests that check connectivity and networking config related best practices. | 4.4.3 |
operator |
The operator test suite is designed to test basic Kubernetes Operator functionality. | 4.4.3 |
platform-alteration |
verifies that key platform configuration is not modified by the CNF under test | 4.4.3 |
Please consult CATALOG.md for a detailed description of tests in each suite.
CNF-specific tests
TODO
Test Output
Claim File
The test suite generates a "claim" file, which describes the system(s) under test, the tests that were run, and the outcome of all of the tests. This claim file is the proof of the test run that is evaluated by Red Hat when "certified" status is being considered. For more information about the contents of the claim file please see the schema. You can read more about the purpose of the claim file and CNF Certification in the Guide.
Adding Test Results for the CNF Validation Test Suite to a Claim File
e.g. Adding a cnf platform test results to your existing claim file.
You can use the claim cli tool to append other related test suite results to your existing claim.json file. The output of the tool will be an updated claim file.
go run cmd/tools/cmd/main.go claim-add --claimfile=claim.json --reportdir=/home/$USER/reports
Args:
--claimfile is an existing claim.json file
--repordir :path to test results that you want to include.
The tests result files from the given report dir will be appended under the result section of the claim file using file name as the key/value pair. The tool will ignore the test result, if the key name is already present under result section of the claim file.
"results": {
"cnf-certification-tests_junit": {
"testsuite": {
"-errors": "0",
"-failures": "2",
"-name": "CNF Certification Test Suite",
"-tests": "14",
Command Line Output
When run the CNF test suite will output a report to the terminal that is primarily useful for Developers to evaluate and address problems. This output is similar to many testing tools.
Test successful output example
Here's an example of a Test pass. It verifies that the CNF is using a replica set:
------------------------------
lifecycle when Testing owners of CNF pod
Should be only ReplicaSet
/Users/$USER/cnf-cert/test-network-function/test-network-function/lifecycle/suite.go:339
2021/07/27 11:41:25 Sent: "oc -n tnf get pods test-697ff58f87-d55zx -o custom-columns=OWNERKIND:.metadata.ownerReferences\\[\\*\\].kind && echo END_OF_TEST_SENTINEL\n"
2021/07/27 11:41:26 Match for RE: "(?s)OWNERKIND\n.+((.|\n)*END_OF_TEST_SENTINEL\n)" found: ["OWNERKIND\nReplicaSet\nEND_OF_TEST_SENTINEL\n" "END_OF_TEST_SENTINEL\n" ""] Buffer: "OWNERKIND\nReplicaSet\nEND_OF_TEST_SENTINEL\n"
•
Test failed output examples
The following is the output from a Test failure. In this case, the test is checking that a CSV (ClusterServiceVersion) is installed correctly, but does not find it (the operator was not present on the cluster under test):
------------------------------
operator Runs test on operators when under test is: my-etcd/etcdoperator.v0.9.4
tests for: CSV_INSTALLED
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122
2020/12/15 15:28:19 Sent: "oc get csv etcdoperator.v0.9.4 -n my-etcd -o json | jq -r '.status.phase'\n"
• Failure [10.002 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:58
Runs test on operators
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:71
when under test is: my-etcd/etcdoperator.v0.9.4
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:121
tests for: CSV_INSTALLED [It]
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122
Expected
<int>: 0
to equal
<int>: 1
The following is the output from a Test failure. In this case, the test is checking that a Subscription is installed correctly, but does not find it (the operator was not present on the cluster under test):
------------------------------
operator Runs test on operators when under test is: my-etcd/etcd
tests for: SUBSCRIPTION_INSTALLED
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:129
2021/04/09 12:37:10 Sent: "oc get subscription etcd -n my-etcd -ojson | jq -r '.spec.name'\n"
• Failure [10.000 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:55
Runs test on operators
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:68
when under test is: default/etcdoperator.v0.9.4
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:128
tests for: SUBSCRIPTION_INSTALLED [It]
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:129
Expected
<int>: 0
to equal
<int>: 1
The following is the output from a Test failure. In this case, the test is checking clusterPermissions for specific CSV, but does not find it (the operator was not present on the cluster under test):
------------------------------
operator Runs test on operators
should eventually be verified as certified (operator redhat-marketplace/etcd-operator)
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:146
• Failure [30.002 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:76
Runs test on operators
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:89
should eventually be verified as certified (operator redhat-marketplace/etcd-operator) [It]
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:146
Timed out after 30.001s.
Expected
<bool>: false
to be true
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:152
Grading Tool
Overview
A tool for processing the claim file and producing a quality grade for the CNF.
The user supplies a policy conforming to policy schema.
A grade is considered passed
if all its direct tests passed and its base grade passed.
In the output we use the field propose
to indicate grade passed or failed.
See policy example for understanding the output of the grading tool.
How to build and execute
make build
or
make build-gradetool
Executable name is gradetool
.
CNF Developers
Developers of CNFs, particularly those targeting CNF Certification with Red Hat on OpenShift, can use this suite to test the interaction of their CNF with OpenShift. If you are interested in CNF Certification please contact Red Hat.
Refer to the rest of the documentation in this file to see how to install and run the tests as well as how to interpret the results.
You will need an OpenShift 4.4 installation running your CNF, and at least one other machine available to host the test suite. The cnf-certification-test-partner repository has a very simple example of this you can model your setup on.
Known Issues
Issue #146: Shell Output larger than 16KB requires specification of the TNF_DEFAULT_BUFFER_SIZE environment variable
When dealing with large output, you may occasionally overrun the default buffer size. The manifestation of this issue is
a json.SyntaxError
, and may look similar to the following:
Expected
<*json.SyntaxError | 0xc0002bc020>: {
msg: "unexpected end of JSON input",
Offset: 660,
}
to be nil
In such cases, you will need to set the TNF_DEFAULT_BUFFER_SIZE to a sufficient size (in bytes) to handle the expected output.
For example:
TNF_DEFAULT_BUFFER_SIZE=32768 ./run-cnf-suites.sh diagnostic generic
ping
or ip
binary utilities
Issue-161 Some containers under test do not contain In some cases, containers do not provide ping or ip binary utilities. Since these binaries are required for the
connectivity tests, we must exclude such containers from the connectivity test suite. In order to exclude these
containers, please add the following to test-network-function/tnf_config.yml
:
excludeContainersFromConnectivityTests:
- namespace: <namespace>
podName: <podName>
containerName: <containerName>
Note: Future work may involve installing missing binary dependencies.