[zuul] Add logging config and tests to functional-autoscaling-tests-osp18

[elfiesmelfie]

Depends-On: #272

Depends-On: openstack-k8s-operators/telemetry-operator#698

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0abffecca8bd45d6b5c9a785d549b469

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 27m 31s
❌ functional-autoscaling-tests-osp18 TIMED_OUT in 3h 13m 18s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 15m 05s
❌ functional-graphing-tests-osp18 FAILURE in 1h 40m 29s (non-voting)

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e2cc7665f2784dadb74303bc395ee688

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 28m 57s
❌ functional-autoscaling-tests-osp18 TIMED_OUT in 3h 13m 22s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 15m 53s
❌ functional-graphing-tests-osp18 FAILURE in 1h 17m 13s (non-voting)

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/442ed5cd2f1a4ce8a358b04f681ec814

❌ functional-autoscaling-tests-osp18 TIMED_OUT in 1h 13m 11s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 15m 55s

[vyzigold]

Found this while doing my round as CI watcher. Interesting. I wonder what's exactly causing the job to take so long. I took a quick look at the must-gather and it looks like logging is enabled and it might even work as far as I can tell. But autoscaling and metricstorage is disabled now. Looking at https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-logging.yml and https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-autoscaling.yml , both define the cifmw_edpm_prepare_kustomizations variable, which basically causes autoscaling to never get enabled. The kustomizations will either need to manually merge or maybe there is some ci-framework magic, which could take 2 kustomizations and apply both of them.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/fd047a41935540b685aacead8486df86

❌ functional-autoscaling-tests-osp18 FAILURE in 4h 33m 48s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 12m 06s

[elfiesmelfie]

recheck

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/982813f0ba1a48de8460662a75ba5e7f

✔️ functional-autoscaling-tests-osp18 SUCCESS in 1h 58m 01s
❌ functional-logging-tests-osp18 FAILURE in 1h 04m 09s

[elfiesmelfie]

Found this while doing my round as CI watcher. Interesting. I wonder what's exactly causing the job to take so long. I took a quick look at the must-gather and it looks like logging is enabled and it might even work as far as I can tell. But autoscaling and metricstorage is disabled now. Looking at https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-logging.yml and https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-autoscaling.yml , both define the cifmw_edpm_prepare_kustomizations variable, which basically causes autoscaling to never get enabled. The kustomizations will either need to manually merge or maybe there is some ci-framework magic, which could take 2 kustomizations and apply both of them.

The main issue is that the logging vars have changed, but the logging job has not been updated to deploy metrics storage or autoscaling. This will eventually be addressed in openstack-k8s-operators/telemetry-operator#698.
This PR was the spike part of the work, where I hack it together with whatever workarounds are needed in order to identify the effort needed to do it properly.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2b0bd6d62ca944ab9d13a9948904e7b5

❌ functional-autoscaling-tests-osp18 FAILURE in 1h 32m 48s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 13m 24s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9b45b588cd9749ff90411c522a15e46c

❌ functional-autoscaling-tests-osp18 RETRY_LIMIT in 3m 18s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 08m 04s

[elfiesmelfie]

recheck

[softwarefactory-project-zuul]

This change depends on a change that failed to merge.

Change openstack-k8s-operators/telemetry-operator#698 is needed.

[elfiesmelfie]

recheck

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/19451bc59e4a4a12ac2a371fb3c8d5a1

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 53m 53s
❌ functional-autoscaling-tests-osp18 POST_FAILURE in 2h 38m 13s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 08m 15s
❌ functional-graphing-tests-osp18 FAILURE in 1h 08m 53s (non-voting)