For some reason, when I do oc logs -n $NAMESPACE $POD -f the command gets stuck after approximately 58 minutes and stops following the logs and outputting anything. The command doesn't crash so it still looks like it's running, but it's not doing anything. This always happens, whether it's started manually or by the script, locally or in the pipeline.
When this happens locally, you can just crash the command with ctrl+c. But when it happens in the pipeline, the pipeline gets aborted after additional 30 minutes and fails. This, I think, happens because of the 30 minutes no-activity timeout set in app-interface.
Here is an example:
15:50:48 +++ bonfire deploy-iqe-cji host-inventory --marker 'not resilience and not cert_auth and not rbac_dependent' --filter ''\'''\''' --image-tag ''\'''\''' --requirements ''\'''\''' --requirements-priority ''\'''\''' --test-importance ''\'''\''' --plugins host_inventory --env clowder_smoke --cji-name host-inventory --namespace ephemeral-b4zal5
15:50:48 2023-01-20 14:50:48 [ WARNING] [ MainThread] split() requires a non-empty pattern match.
15:50:50 2023-01-20 14:50:50 [ INFO] [ MainThread] processing IQE ClowdJobInvocation
15:50:50 2023-01-20 14:50:50 [ INFO] [ MainThread] running (pid 2231426): oc apply -f - -n ephemeral-b4zal5
15:50:51 2023-01-20 14:50:51 [ INFO] [ pid-2231426] |stdout| clowdjobinvocation.cloud.redhat.com/host-inventory created
15:50:51 2023-01-20 14:50:51 [ INFO] [ MainThread] waiting on CJI 'host-inventory' for max of 600sec...
15:50:51 2023-01-20 14:50:51 [ INFO] [ MainThread] waiting for Job to appear owned by CJI 'host-inventory'
15:50:51 2023-01-20 14:50:51 [ INFO] [ MainThread] found Job 'host-inventory-iqe-25occ8v' created by CJI 'host-inventory', now waiting for pod to appear
15:50:51 2023-01-20 14:50:51 [ INFO] [ MainThread] found pod 'host-inventory-iqe-25occ8v-sgfc7' associated with CJI 'host-inventory', now waiting for pod to be 'running'
15:50:52 2023-01-20 14:50:52 [ INFO] [ MainThread] [pod/host-inventory-iqe-25occ8v-sgfc7] waiting up to 599sec for resource to be 'ready'
15:50:57 2023-01-20 14:50:57 [ INFO] [ MainThread] [pod/host-inventory-iqe-25occ8v-sgfc7] resource is ready!
15:50:57 2023-01-20 14:50:57 [ INFO] [ MainThread] pod 'host-inventory-iqe-25occ8v-sgfc7' related to CJI 'host-inventory' in ns 'ephemeral-b4zal5' is running
15:50:57 ++ POD=host-inventory-iqe-25occ8v-sgfc7
15:50:57 ++ set +x
15:50:57 ++ oc_wrapper wait --timeout=3h --for=condition=JobInvocationComplete -n ephemeral-b4zal5 cji/host-inventory
15:51:43 Plugin path: /iqe_venv/lib/python3.8/site-packages/iqe_host_inventory
15:51:43
15:51:43 ---------------------------- live log sessionstart -----------------------------
15:51:43 2023-01-20 14:51:01 INFO Successfully authenticated to vault: https://vault.ci.ext.devshift.net/
.
.
.
16:49:49 2023-01-20 15:49:49 INFO MessageWrapper(_MessageWrapper__message=<cimpl.Message object at 0x7fb7bdc317c0>, _MessageWrapper__overidden_value=None, _MessageWrapper__overidden_host=None).value()
16:49:49 2023-01-20 15:49:49 INFO MessageWrapper(_MessageWrapper__message=<cimpl.Message object at 0x7fb7bdc317c0>, _MessageWrapper__overidden_value=None, _MessageWrapper__overidden_host=HostWrapper({ 'account': None,
17:19:49 'ansible_host': 'web-50.lewis-willis.info',Build timed out (after 30 minutes). Marking the build as failed.
17:19:49 Build was aborted
17:19:49 Recording test results
17:19:49 ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
17:19:49 Archiving artifacts
17:19:49 Terminated
Look at the timestamps for the beginning of the test run (15:51:43), the last returned log message (16:49:49) and the time when the job was aborted (17:19:49). This is not an issue with the wrong oc wait command, the IQE_CJI_TIMEOUT is set to 3 hours and you can also see that in the logs, so it is the oc logs -f command that's not behaving correctly.
I don't know why the logs stop being followed after 1 hour, if there is some timeout or if it is the amount of data, but if you would run that command again it would continue for another hour without any problems.
I think the cji_smoke_test.sh should account for this and have some workaround because otherwise, it is not usable for test runs longer than 1 hour.
For some reason, when I do
oc logs -n $NAMESPACE $POD -fthe command gets stuck after approximately 58 minutes and stops following the logs and outputting anything. The command doesn't crash so it still looks like it's running, but it's not doing anything. This always happens, whether it's started manually or by the script, locally or in the pipeline.When this happens locally, you can just crash the command with ctrl+c. But when it happens in the pipeline, the pipeline gets aborted after additional 30 minutes and fails. This, I think, happens because of the 30 minutes no-activity timeout set in app-interface.
Here is an example:
Look at the timestamps for the beginning of the test run (15:51:43), the last returned log message (16:49:49) and the time when the job was aborted (17:19:49). This is not an issue with the wrong
oc waitcommand, theIQE_CJI_TIMEOUTis set to 3 hours and you can also see that in the logs, so it is theoc logs -fcommand that's not behaving correctly.I don't know why the logs stop being followed after 1 hour, if there is some timeout or if it is the amount of data, but if you would run that command again it would continue for another hour without any problems.
I think the
cji_smoke_test.shshould account for this and have some workaround because otherwise, it is not usable for test runs longer than 1 hour.