Stability issues wrt Scheduled Benchmark Tests on Self Hosted Runner

[hershd23]

Description

Multiple issues of the Scheduled Benchmark Tests on Self Hosted Runner generate different results

While here just one combination failed :- https://github.com/layer5io/meshery-smp-action/runs/7770247406 (linkerd-wrk-soak)

Here a couple failed :- https://github.com/layer5io/meshery-smp-action/actions/runs/2832186044

And here one runner startup itself failed :- https://github.com/layer5io/meshery-smp-action/actions/runs/2833335073

Expected Behavior

The behaviour should be consistent across Scheduled Benchmark Test run on self hosted runners. Given we are working with external hardware where connections can fail, ideally we should have a retry mechanism and we should work to considerably reduce the consistency

Logs

In the logs Auth seems to be a common culprit

Opening Meshery (http://localhost:31391) in browser.
Failed to open Meshery in browser, please point your browser to http://localhost:31391 to access Meshery.
authentication failed: Get "http://localhost:31[39](https://github.com/layer5io/meshery-smp-action/runs/7765562299?check_suite_focus=true#step:5:40)1/api/providers": dial tcp [::1]:31391: connect: connection refused
Verifying prerequisites...
Authentication token not found. please supply a valid user token with the --token (or -t) flag. or login with `mesheryctl system login`
Onboarding application... Standby for few minutes...
Error: Authentication token not found. please supply a valid user token with the --token (or -t) flag. or login with `mesheryctl system login`

This is seen for istio

Opening Meshery (http://192.168.49.2:32398/) in browser.
Failed to open Meshery in browser, please point your browser to http://192.168.49.2:32398/ to access Meshery.
Verifying prerequisites...
Adapter for required mesh not found
Onboarding application... Standby for few minutes...
rpc error: code = Unknown desc = no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"

Error from server (NotFound): namespaces "istio-system" not found
Error from server (NotFound): namespaces "istio-system" not found
Service Mesh: Istio - ISTIO
Gateway URL: [http://192.168.49.2:](http://192.168.49.2/)

Environment:

• Meshery Version: v0.6.0-rc.6fc
• Kubernetes Version: minikube 1.23.2
• Host OS: Ubuntu
• Browser: N/A

[hershd23]

@gyohuangxin @leecalcote

[leecalcote]

@hershd23 thanks for this! @alphaX86 will you kindly investigate the auth issue? // @theBeginner86

[alphaX86]

Rog' seems I need to see the script file again and tackle the auth issue

[hershd23]

https://github.com/layer5io/meshery-smp-action/tree/master/.github/workflows/scripts @alphaX86

[hershd23]

Just to point out the auth issue doesn't happen everytime, it's flaky but this seems to be one of the most clear reasons as to why random tests are failing

[alphaX86]

Understood, I kinda figured where it went wrong, but still I need to investigate further...

The fix for this will be done by tomorrow for sure. Thanks @hershd23 👍

[hershd23]

Great. Do let me know if the scripts need to change in lieu of your changes to meshery or ctl