[FAILED] in [BeforeAll] - /src/test/plugins/slo/slo_test.go:211 @ 08/10/23 17:07:44.151
  2023-08-10T17:07:44Z INFO plugin.alerting exiting main sync loop
  2023-08-10T17:07:44Z INFO plugin.alerting shutting down cluster driver update handler
  2023-08-10T17:07:44Z WARN jetstream disconnected from jetstream
  2023-08-10T17:07:44Z WARN jetstream disconnected from jetstream
  2023-08-10T17:07:44Z DEBUG plugin.alerting closing cortex cluster status watcher... {"watcher": "cortex-cluster-status"}
  2023-08-10T17:07:44Z ERROR plugin.alerting failed to receive cluster event : rpc error: code = Canceled desc = context canceled
  2023-08-10T17:07:44Z DEBUG plugin.metrics rule discovery stream closing {"error": "context canceled"}
  2023-08-10T17:07:44Z INFO gateway shutting down plugins
  2023-08-10T17:07:44Z WARN plugin.metrics rule discovery stream closing {"error": "context canceled"}
  2023-08-10T17:07:44Z WARN plugin.metrics rule discovery stream closing {"error": "context canceled"}
  2023-08-10T17:07:44Z WARN jetstream disconnected from jetstream
  2023-08-10T17:07:44Z DEBUG plugin.metrics rule discovery stream closing {"error": "context canceled"}
  2023-08-10T17:07:44Z DEBUG gateway.sync agent disconnected {"id": "agent2"}
  2023-08-10T17:07:44Z INFO gateway.grpc agent stream closing {"error": "context canceled"}
  2023-08-10T17:07:44Z DEBUG gateway.sync agent disconnected {"id": "agent"}
  2023-08-10T17:07:44Z DEBUG gateway.delegate agent disconnected {"id": "agent2"}
  2023-08-10T17:07:44Z INFO gateway.grpc agent stream closing {"error": "context canceled"}
  2023-08-10T17:07:44Z DEBUG gateway.delegate agent disconnected {"id": "agent"}
  2023-08-10T17:07:44Z INFO test.env Cortex stopping...
  2023-08-10T17:07:44Z WARN plugin.alerting failed to receive cluster health status from grpc stream, retrying...
  2023-08-10T17:07:44Z WARN gateway http server exited with error {"error": "context canceled"}
  2023-08-10T17:07:44Z WARN test.env gateway server exited with error {"error": "context canceled"}
  2023-08-10T17:07:44Z ERROR plugin.alerting EOF
  2023-08-10T17:07:44Z INFO gateway all plugins shut down
  2023-08-10T17:07:44Z WARN plugin.alerting/pkg/alerting.stream stream server exited with error {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN plugin.alerting/pkg/alerting.stream stream server exited with error {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN plugin.metrics/pkg/gateway.stream stream server exited with error
  2023-08-10T17:07:44Z WARN plugin.metrics/pkg/gateway.stream stream server exited with error {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN agent disconnected from gateway {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN agent shutting down gateway client {"error": "context canceled"}
  2023-08-10T17:07:44Z WARN plugin.metrics/pkg/agent.stream stream disconnected with error
  2023-08-10T17:07:44Z ERROR test agent "agent" exited: context canceled
  2023-08-10T17:07:44Z WARN test.env management server exited with error {"error": "context canceled"}
  2023-08-10T17:07:44Z WARN plugin.metrics/pkg/agent.stream stream disconnected with error {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN plugin.alerting/pkg/agent.stream stream disconnected with error {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN agent disconnected from gateway {"error": "rpc error: code = Unavailable desc = error reading from server: read tcp 127.0.0.1:40786->127.0.0.1:34317: use of closed network connection"}
  2023-08-10T17:07:44Z WARN plugin.alerting/pkg/agent.stream stream disconnected with error {"error": "rpc error: code = Canceled desc = context canceled"}
  2023-08-10T17:07:44Z WARN agent shutting down gateway client {"error": "context canceled"}
  2023-08-10T17:07:44Z ERROR test agent "agent2" exited: context canceled
  2023-08-10T17:07:44Z INFO test.env Cortex stopped
  << Timeline

  [FAILED] Timed out after 30.001s.
  Expected success, but got an error:
      <*status.Error | 0xc0020e0818>: 
      rpc error: code = Unknown desc = failed to get cluster stats: 500
      {
          s: {
              s: {
                  state: {
                      NoUnkeyedLiterals: {},
                      DoNotCompare: [],
                      DoNotCopy: [],
                      atomicMessageInfo: nil,
                  },
                  sizeCache: 0,
                  unknownFields: nil,
                  Code: 2,
                  Message: "failed to get cluster stats: 500",
                  Details: nil,
              },
          },
      }
  In [BeforeAll] at: /src/test/plugins/slo/slo_test.go:211 @ 08/10/23 17:07:44.151
------------------------------
SSSSSSSSSSS
------------------------------
P [PENDING]
Converting ServiceLevelObjective Messages to Prometheus Rules when CRUDing SLOs Should preview SLOs in a raw data format [integration, slow]
/src/test/plugins/slo/slo_test.go:479
------------------------------
P [PENDING]
Converting ServiceLevelObjective Messages to Prometheus Rules when CRUDing SLOs Creating an SLO for the service that should be alerting [integration, slow]
/src/test/plugins/slo/slo_test.go:567
------------------------------
S

Summarizing 1 Failure:
  [FAIL] Converting ServiceLevelObjective Messages to Prometheus Rules [BeforeAll] when The instrumentation server starts Should simulate events [integration, slow]
  /src/test/plugins/slo/slo_test.go:211

Ran 1 of 15 Specs in 30.677 seconds
FAIL! -- 0 Passed | 1 Failed | 2 Pending | 12 Skipped
--- FAIL: TestSloPlugin (30.68s)
FAIL
FAIL	github.com/rancher/opni/test/plugins/slo	31.000s
ok  	github.com/rancher/opni/test/web	1.187s
FAIL
Stderr:
Running target: Test:All
exec: go "test" "-race" "./..."
Error: running "go test -race ./..." failed with exit code 1
Please visit https://dagger.io/help#go for troubleshooting guidance.
exit status 1