feat(vector): implement spooling logic

[JoshFerge]

Right now when a vector request fails, we simply drop the uptime check result. This is not acceptable. The following PR does the following:

• Introduces two new config items retry_vector_errors_forever

If retry_vector_errors_forever is set to True, we will retry making the vector request for a batch forever if the request fails due to a non 2XX or other error. Messages will spool on the channel in the meantime. Given the current machine size and requests per second a pod handles, this should give us roughly 2.5 days of spooling before the service fails from an OOM. (assuming 500 byte check result size, and 4GB pod memory limit).

We also assume for the sake of this idea that we will produce vector results faster than we will uptime checks. I assume this a safe assumption, although if vector is having problems, the queue could still increase while we're producing successfully to vector.

• Introduces a metric for vector request worker queue size, and vector response time.

We can create monitors on both of these metrics, and an aggressive queue size monitor should allow for plenty of time for the on-call to take action before a queue fills up & process dies.

---

In a future PR, we could consider a strategy like dropping all successful uptime checks, if the queue is too full. can evaluate options with the team

[wedamija]

To some extent I wonder if it's even worth having the max number of retries, if we have days of spooling buffer available

[evanpurkhiser]

nit: Having this whoel block inside the if is getting a bit unweildy, I think we could just do the if on the result itself

if let Err(e) = result {
   // ...
}

[evanpurkhiser]

Looks good. I think we could clean up some of the metric and logs after in this module to make them all more consistent maybe