|
| 1 | +# Cross-component E2E for operator framework |
| 2 | + |
| 3 | +This is a cross-component demo with all OLM v1 repositories. The ginkgo test does the following: |
| 4 | +- Uses operator-sdk and kustomize to build `plain+v0` bundles and create catalogs to include the bundles. |
| 5 | +- Installs, upgrades and deletes a `plain+v0` operator. |
| 6 | +- Uses operator-sdk to build `registry+v1` bundles and create catalogs to include the bundles. |
| 7 | +- Installs, upgrades and deletes a `registry+v1` operator. |
| 8 | + |
| 9 | +The steps in the ginkgo test can be summarized as follows: |
| 10 | + |
| 11 | +1. start with an empty directory |
| 12 | +2. call operator-sdk to initialize and generate an operator |
| 13 | +3. generate a bundle directory |
| 14 | +4. build/push/kind load bundle images from the bundle directories |
| 15 | +5. repeat steps 2-4 as necessary to get bundles for multiple operator versions |
| 16 | +6. generate a catalog directory |
| 17 | +7. build/push/kind load the catalog |
| 18 | +8. create a Catalog CR (with kubectl operator) |
| 19 | +9. create an Operator CR (with kubectl operator) |
| 20 | +10. trigger Operator upgrades (with kubectl operator) |
| 21 | +11. delete the Operator CR (with kubectl operator) |
| 22 | +12. delete the Catalog CR (with kubectl operator) |
| 23 | +13. repeat steps 2-12 for each bundle format (e.g. registry+v1 and plain+v0) |
| 24 | +## Objective |
| 25 | +- Development on OLM v1 is split across multiple repositories, and the list of relevant repositories may grow over time. While we have demos showing improvements in functionality of components over time, it can be difficult to have a picture of the state of OLM v1 at any given time for someone not following its development closely. Having a single source to look for OLM v1 behavior can provide more clarity about the state of the project. |
| 26 | +- With the scale of the OLM v1 project, it is useful to have a means to test components in the operator development + lifecycle pipeline together to create a more cohesive experience for all users. |
| 27 | + |
| 28 | +## Getting Started |
| 29 | +- This test currently only works with the container runtime `docker`. |
| 30 | +- Building operator-controller, deploying it into the cluster and rest of the configuration is done in the `MakeFile` of this repo under the target `operator-developer-e2e`. This includes: |
| 31 | + |
| 32 | + - Setting up a kind cluster named `operator-controller-op-dev-e2e`. |
| 33 | + - Installing the operator controller onto the cluster. |
| 34 | + - Setting up `opm`, `operator-sdk` and `kustomize` using bingo. |
| 35 | + - Setting up a local registry server for building and loading images. |
| 36 | +### Input Values used |
| 37 | + |
| 38 | +Below are the input values used in the test which is specified in the `operator_framework_test.go`. |
| 39 | + |
| 40 | +- The following structs defined are required, to accept input for both `plain+v0` and `registry+v1` bundles: |
| 41 | + - For getting bundle related inputs: |
| 42 | + ``` |
| 43 | + type BundleInfo struct { |
| 44 | + baseFolderPath string |
| 45 | + bundles []BundleContent |
| 46 | + } |
| 47 | + |
| 48 | + type BundleContent struct { |
| 49 | + bInputDir string |
| 50 | + bundleVersion string |
| 51 | + imageRef string |
| 52 | + } |
| 53 | + ``` |
| 54 | + - `baseFolderPath` - Base/root path of the folder where the specific bundle type input data is stored.[root path to plain-v0 or registry-v1 bundles testdata] |
| 55 | + - `bundles` - Stores the data relevant to different versions of the bundle. |
| 56 | + - `bInputDir` - The directory that stores the specific version of the bundle data. The name of the directory is formed and is of the format `<operatorName>.v<bundleVersion>`. |
| 57 | + - `bundleVersion` - The specific version of the bundle data. |
| 58 | + - `imageRef` - This is formed. Stores the bundle image reference which will be of the format `<registry_repo>/< operatorName>-bundle:v.<bundleVersion>` |
| 59 | + - For getting catalog related inputs: |
| 60 | + ``` |
| 61 | + type CatalogDInfo struct { |
| 62 | + baseFolderPath string |
| 63 | + catalogDir string |
| 64 | + operatorName string |
| 65 | + desiredChannelName string |
| 66 | + imageRef string |
| 67 | + fbcFileName string |
| 68 | + } |
| 69 | + ``` |
| 70 | + - `baseFolderPath` - Base/root path of the folder that stores the catalogs. |
| 71 | + - `operatorName` - Name of the operator to be installed from the bundles. |
| 72 | + - `desiredChannelName` - Desired channel name for the operator. |
| 73 | + - `catalogDir` - This is formed. The directory to store the catalog/FBC. The directory name will be of the format: `<operator-name>-catalog` |
| 74 | + - `imageRef` - This is formed. Stores the FBC image reference which will be of the format: `<registry_repo>/<catalogDir>:test` |
| 75 | + - `fbcFileName` - Name of the FBC file. This is hard-coded as `catalog.yaml`. |
| 76 | + - For getting information related to the install/upgrade action for operators: |
| 77 | + ``` |
| 78 | + type OperatorActionInfo struct { |
| 79 | + installVersion string |
| 80 | + upgradeVersion string |
| 81 | + } |
| 82 | + ``` |
| 83 | + - `installVersion` - Version of the operator to be installed on the cluster. |
| 84 | + - `upgradeVersion` - Version of the operator to be upgraded on the cluster. |
| 85 | +
|
| 86 | + - The below inputs are used to form the bundle using operator-sdk. |
| 87 | +
|
| 88 | + ``` |
| 89 | + type SdkProjectInfo struct { |
| 90 | + projectName string |
| 91 | + domainName string |
| 92 | + group string |
| 93 | + version string |
| 94 | + kind string |
| 95 | + } |
| 96 | + ``` |
| 97 | +## How to run |
| 98 | +- Makefile target `operator-developer-e2e` : Runs the entire E2E setup. |
| 99 | +- Makefile target `test-op-dev-e2e`: Runs the ginkgo test. |
| 100 | +- Makefile target `deploy-local-registry`: Deploys local registry |
| 101 | +- Makefile target `cleanup-local-registry` : Stops and removes local registry |
| 102 | +- Makefile target `kind-cluster-cleanup`: Deletes the kind cluster |
| 103 | +
|
| 104 | +## Bundle Types |
| 105 | +### Plain bundles |
| 106 | +- The `plain+v0` bundles are formed using `operator-sdk` and `kustomize`. |
| 107 | + - The `kustomize` organizes the different resources present in the `operator-sdk` project into a single yaml file. |
| 108 | + - The Dockerfile for the bundle is named `plainbundle.Dockerfile` and is generated using a custom routine. |
| 109 | + - The generated bundle is stored in the format: |
| 110 | + ``` |
| 111 | + plain-v0 |
| 112 | + └── <operatorName>.v<bundleVersion> |
| 113 | + └── manifests |
| 114 | + │ └── mainfest.yaml |
| 115 | + └── plainbundle.Dockerfile |
| 116 | + ``` |
| 117 | +
|
| 118 | +
|
| 119 | +- The FBC template is formed by a custom routine by using the operatorName, desiredChannelName, bundle imageRefs and bundleVersions. |
| 120 | + - `Default channel` is not used in forming the FBC as it is not an OLMv1 concept. |
| 121 | + - Only one `olm.channel` is generated which is the given <desiredChannelName>. |
| 122 | + - Upgrade graph is formed with only replaces edge. |
| 123 | + - The generated FBC is not validated using `opm` as the tool has no support for plain bundles. |
| 124 | + - The Dockerfile for the catalog is named `<operator-name>-catalog.Dockerfile` and is generated using a custom routine. |
| 125 | + - The generated catalog is stored in the format: |
| 126 | + ``` |
| 127 | + catalogs |
| 128 | + └── <operator-name>-catalog |
| 129 | + │ └── catalog.yaml |
| 130 | + └── <operator-name>-catalog.Dockerfile |
| 131 | + ``` |
| 132 | +- The catalog CR is then formed with the name `<operatorName>-catalog`. |
| 133 | +
|
| 134 | +- The operator is then installed and has the name `<operatorName>`. |
| 135 | +
|
| 136 | +### Registry Bundles |
| 137 | +
|
| 138 | +- The registry+v1 bundles are formed using operator-sdk. |
| 139 | + - The generated CSV uses the default values. |
| 140 | + - The bundle content is formed within the operator-sdk project directory in the folder `bundle`. This is moved to the bundle directory folder. |
| 141 | + - The generated Dockerfile has the name `bundle.Dockerfile`. The Dockerfile and bundle structure is genearted by the `operator-sdk` tool. |
| 142 | + - The generated bundle is stored in the format: |
| 143 | + ``` |
| 144 | + registry-v1 |
| 145 | + └── <operatorName>.v<bundleVersion> |
| 146 | + └── manifests |
| 147 | + └── metadata |
| 148 | + └── bundle.Dockerfile |
| 149 | + ``` |
| 150 | +
|
| 151 | +- The FBC is formed using `opm alpha render-template semver` tool. |
| 152 | + - The semver template named `registry-semver.yaml` is formed using a custom routine by passing the bundle imageRefs. |
| 153 | + - `generatemajorchannels` and `generateminorchannels` is set to false in the semver template. |
| 154 | + - The generated catalog is stored in the format: |
| 155 | + ``` |
| 156 | + catalogs |
| 157 | + └── <operator-name>-catalog |
| 158 | + │ └── catalog.yaml |
| 159 | + └── <operator-name>-catalog.Dockerfile |
| 160 | + ``` |
| 161 | +
|
| 162 | +- The catalog resource is then formed with the name `<operatorName>-catalog`. |
| 163 | +
|
| 164 | +- The operator is then installed and has the name `<operatorName>`. |
| 165 | +
|
| 166 | +
|
| 167 | +- After the e2e workflow, all the files formed are cleared. |
| 168 | +
|
| 169 | +
|
| 170 | +## To-do |
| 171 | +1. The resources are read from input manifests using universal decoder from `k8s.io/apimachinery/pkg/runtime/serializer`. |
| 172 | + - However, in cases where a single file contains multiple YAML documents separated by `---,` the `UniversalDecoder` recognizes only the first resource. This situation is relevant as for `plain+v0` bundles generated through `kustomize,` the manifest has multiple YAML documents are combined into one file using --- separators. This is now handled by splitting the content of the YAML file and decoding each of them using the `UniversalDecoder`. |
| 173 | + - This workaround can be improved using `YAMLToJSONDecoder` from `k8s.io/apimachinery/pkg/util/yaml`. And the kind, api version and name can be get by decoding into `Unstructured` from `k8s.io/apimachinery/pkg/apis/meta/v1/unstructured`. |
| 174 | +2. All the tests pass and the operator is installed successfully. The bundledeployment succeeds and the resources are created. But the pod for the new operator failes due to `ImagePullBackOff`. |
| 175 | + - This is because the `Deployment` controller-manager uses the image `controller:latest` which is not present on the cluster. |
| 176 | + - The solution would be to replace `controller:latest` with the `busybox:latest` and then pulling and loading `busybox:latest` onto cluster. |
| 177 | + - The replacement could possibly be achieved by adding the following to `config/default/kustomization.yaml` under `operator-sdk` project: |
| 178 | + ``` |
| 179 | + images: |
| 180 | + - name: controller |
| 181 | + newName: controller |
| 182 | + newTag: latest |
| 183 | + ``` |
| 184 | +## Issues |
| 185 | +1. This test currently only works with the container runtime `docker`. |
| 186 | + - The default value of CONTAINER_RUNTIME defined in the Makefile is `docker`. Therefore the correct runtime has to be assigned to the variable `CONTAINER_RUNTIME` before calling the make target if docker is what is not being used. The test routine also assumes the runtime as `docker` if it is unable to retrieve the value of the environment variable. |
| 187 | + - But this is only a partial fix to the problem. With this change the test for `plain+v0` bundles will pass but for `registry+v1` will fail for other container runtimes. This is because `registry+v1` uses `operator-sdk` support. Thus to mimic the user experience, the targets `bundle-build` and `bundle-push` from the generated Makefile by operator-sdk tool, which has docker being hard coded as the container runtime, is used to build and push the bundle images. This could be marked as an issue and addressed when hard coding docker as container runtime in the generated Makefile is addressed by operator-sdk. |
| 188 | +
|
| 189 | +2. The `opm`,`operator-sdk` and `kustomize` binaries are added in operator-controller using `bingo`. |
| 190 | + - But based on discussions, if required test should be changed so that it has `opm` and `operator-sdk` in `PATH` and simply runs it like `exec.Command("opm", ...)`. |
| 191 | + - This will enable in running [a matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) for the tests and to use different versions of `opm` and `operator-sdk`. |
| 192 | + - This might help in emulating the user experience better. |
| 193 | +
|
| 194 | +## Tooling gaps |
| 195 | +
|
| 196 | +Following are the tooling gaps identified while testing `operator-framework` end-to-end: |
| 197 | +- `opm` doesn't have plain bundle support. |
| 198 | +- No tool for forming FBC for plain bundles. |
| 199 | +- No tool for generating Dockerfile for plain bundles. |
| 200 | +- No tool for generating Dockerfile for plain catalogs. |
| 201 | +- Since `opm` doesn't have plain bundle support, there is no means to validate the FBC generated for plain bundles. |
| 202 | + |
0 commit comments