Mark test servers for auto-deletion using "end_at" parameter

[leecalcote]

Current Behavior

Of the scheduled tests that run multiple times a day, they have faced a few challenges. Notably, one of those challenges is in the cleanup phase once a test is complete. Currently, it is frequently the case that any number of bare metal servers that are used for testing or orphaned, and not decommissioned at the end of each test. This leaves an inordinate amount of bare metal servers, unnecessarily unavailable for used by other projects.

@vielmetti has been most helpful in identifying ways to mitigate this from happening.

Desired Behavior

All resources provisioned for a scheduled test are subsequently decommissioned at the end of that same test.

Implementation

Recently @vielmetti point this out:

You can create servers that will auto-delete themselves at a time certain, perfect for test runs. See https://deploy.equinix.com/developers/docs/metal/deploy/spot-market/#spot-market-request-creation. You want the “end_at” parameter on the API endpoint for device creation

Slack Message

Acceptance Tests

1. Test servers are decommissioned at end of scheduled testing period.

---

Contributor Guide

[vielmetti]

looks like it's here that needs to be changed, and @gyohuangxin had the last edit. (actually looks like a pretty simple fix). We'll need to compute a timestamp.

.github/workflows/scripts/start-cil-runner.sh

What I don't know is, how long are the tests expected to run, at worst case? Can the machines be clobbered in 24, 6, 2 hours?

[gyohuangxin]

@vielmetti Currently, it runs in 30mins, at worst case. But considering that we may add more test cases in the future, I think 2 hours is a appropriate deadline.

[vielmetti]

@gyohuangxin The Equinix system bills by the hour, so my recommendation is to set the expiry at say 1 hour 50 minutes from the time it's created, that will catch anything relatively fast but not risk running 2:01 and incurring the extra charge.

The example end_at time is in ISO format, e.g

"end_at": "2020-09-24T05:00:00Z",

which can be generated with something like date -v +110M -Iminutes .

[gyohuangxin]

@vielmetti I create a PR to fix this: https://github.com/layer5io/meshery-smp-action/pull/76/files, but I want to confirm the default time zone used by the Equinix system:
https://github.com/layer5io/meshery-smp-action/pull/76/files#diff-0e9a22dd7a8bbc870d2ab77c5392deb500a1f946d0e6ff615d75792a2cd8e977R20

[gyohuangxin]

@vielmetti I confirmed that Equinix system is using UTC/GMT timezone, so please review the PR, thanks.

[vielmetti]

Thanks @gyohuangxin - I am checking on the semantics of "termination_time", to see if this is expected to work for ordinary instances or only for "spot instances". Will follow up soonest when I can confirm.

[vielmetti]

Confirming two things: one, that our team is working on updated docs for the termination_time option, and two, that based on my understanding this should work as planned.

[vielmetti]

To complete this -

We have updated the Equinox Metal "termination_time" docs at https://deploy.equinix.com/developers/api/metal/#tag/Devices/operation/createDevice to reflect better the use case (ephemeral instances) and to document the time zone question described above.

Since this change was deployed last month we have not had any of the previous issues described! That's all good news.