example-cnf | Document other SRIOV setups and enable privileged mode

Author: ramperher

roles/example_cnf_deploy/README.md

@@ -45,6 +45,7 @@ Also, exported `kubeconfig` file is required to run the Ansible tasks on the tar
 | ecd_networks_cnfapp                   | []                                                                                              | no       | Networks for CNFApp, including MAC addresses and (if provided) IP addresses |
 | ecd_networks_trex                     | []                                                                                              | no       | Networks for TRex, including MAC addresses and (if provided) IP addresses |
 | ecd_run_deployment                    | 1                                                                                               | no       | Run all deployment automation. If different than 1, the automation will only create the pods and prepare the scripts to launch testpmd/grout and trex manually afterwards |
+| ecd_enable_privileged_mode            | false                                                                                           | no       | Enable container privileged mode, instead of setting specific capabilities. This is required when using Mellanox cards. As it uses netdevice instead of vfio-pci, it requires access to /dev/vfio/vfio from host. Using a volume with hostPath is not enough to achieve it (see [this](https://access.redhat.com/solutions/6560521) for an example). |
 | ecd_termination_grace_period_seconds  | 30                                                                                              | no       | Termination grace period for TestPMD |
 | ecd_testpmd_reduced_mode              | 0                                                                                               | no       | Use reduced mode for TestPMD (if different than 0), where only three cores are used, and txd/rxd parameters are doubled |
 | ecd_trex_test_config                  | []                                                                                              | no       | TRex test configuration. See an example below |
@@ -112,6 +113,10 @@ to:
       capabilities: '{"mac": true}'
 ```
 
+This configuration is related to Intel cards. If you are using other type of cards, you need to apply the corresponding modifications (`device_id` and `vendor` will change, among others). For example, [for Mellanox cards](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/networking/hardware-networks#example-vf-use-in-dpdk-mode-mellanox_using-dpdk-and-rdma), you need to use `netdevice` instead of `vfio-pci`, and also enable RDMA (`is_rdma: true`).
+
+> Remember that Mellanox requires containers to run in privileged mode. Enable it with `ecd_enable_privileged_mode: true`.
+
 ## Network configuration
 
 To apply a proper network configuration for both CNFApp and TRex, you need to follow a YAML structure like this, to be provided in a file pointed by `ecd_network_config_file` (e.g. `/path/to/net-config.yml`). This is just required when using Grout, since TestPMD acts in MAC forwarding mode, so no IP addresses are required for it (but you can define them anyway, only thing is that they're not going to be considered for traffic forwarding):

roles/example_cnf_deploy/defaults/main.yml

@@ -67,6 +67,12 @@ ecd_networks_trex: []
 # to launch testpmd/grout and trex manually afterwards
 ecd_run_deployment: 1
 
+# Enable container privileged mode, instead of setting specific capabilities.
+# This is required when using Mellanox cards. As it uses netdevice instead of vfio-pci,
+# it requires access to /dev/vfio/vfio from host. Using a volume with hostPath is not
+# enough to achieve it (see https://access.redhat.com/solutions/6560521 for an example).
+ecd_enable_privileged_mode: false
+
 # RuntimeClass that should be used for running DPDK application,
 # if the var is empty, the annotation irq-load-balancing.crio.io: "disable" is not applied
 # ecd_high_perf_runtime: "performance-blueprint-profile"

roles/example_cnf_deploy/templates/grout-cr.yaml.j2

@@ -4,7 +4,7 @@ metadata:
   name: grout
   namespace: {{ ecd_cnf_namespace }}
 spec:
-  privileged: false
+  privileged: {{ ecd_enable_privileged_mode }}
   imagePullPolicy: {{ ecd_image_pull_policy }}
   size: 1  
   networks: {{ ecd_networks_cnfapp }}

roles/example_cnf_deploy/templates/testpmd-cr.yaml.j2

@@ -4,7 +4,7 @@ metadata:
   name: testpmd
   namespace: {{ ecd_cnf_namespace }}
 spec:
-  privileged: false
+  privileged: {{ ecd_enable_privileged_mode }}
   imagePullPolicy: {{ ecd_image_pull_policy }}
   ethpeerMaclist: {{ ecd_trex_mac_list }}
   size: 1  

roles/example_cnf_deploy/templates/trex-server-cr.yaml.j2

@@ -16,6 +16,7 @@ spec:
   imagePullPolicy: {{ ecd_image_pull_policy }}
   networks: {{ ecd_networks_trex }}
   cpu: 6
+  privileged: {{ ecd_enable_privileged_mode }}
   runDeployment: {{ ecd_run_deployment }}
 {% if ecd_trex_config_skip|default(false)|bool %}
   trex_config_skip: true