Deploy new monitoring stack

[upodroid]

Which issue(s) this PR fixes:

Part of #3253

Dashboard available at
https://console.cloud.google.com/monitoring/dashboards/builder/01537922-706c-4ab6-8550-c1aa1cd25910?authuser=1&orgonly=true&project=knative-tests&supportedpurview=project&dashboardBuilderState=%257B%2522editModeEnabled%2522:false%257D&timeDomain=1h

/cc @kvmware

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[krsna-m]

Thanks for tackling this one. With these new changes we probably don't need https://github.com/knative/test-infra/tree/main/prow/cluster/monitoring anymore and that can be safe to remove.

[krsna-m]

cc @chizhg if you are interested :)

[upodroid]

With these new changes we probably don't need https://github.com/knative/test-infra/tree/main/prow/cluster/monitoring anymore and that can be safe to remove.

I'll remove it in a separate PR. I'm not quite happy with the Managed Prometheus and GCP Monitoring integration. I'll most likely end up deploying a public grafana instance with a selection of those dashboards.

[krsna-m]

+1 to the hosted monitoring. There was some concern about the managed price.
/lgtm

[upodroid]

+1 to the hosted monitoring. There was some concern about the managed price.

I'll keep an eye on the costs. Managed Prometheus pricing was discounted recently. (IMHO it was overpriced and I was part of the private preview). It saves a lot of operational work with running Prometheus.