Goroutines return value in agent.

[flashvoid]

So, our handlers (e.g. index) returns success asynchronously, after spawning processing thread. What happens if processing thread failed - well it returns an error but there is no one to act on it.

If processing thread will die in a middle of execution leaving half configured resource, i suggest we implement some slice of channels and will spin up goroutine that would poll these channels for return values from our worker.

There is probably not much we can really do once we discover an error but we should at least make some noise about it.

UPD: some new thoughts coming up, Pritesh mentioned nova notifier that can fail a VM but we'd have different backends and putting support for all the possible APIs out there smells bad. I suggest we implement something like hooks - arbitrary executables configured via commandline and called when needed (e.g. ./hooks/fail "token" "reason").

For different backend we can just swap hooks.

[jbrendel]

I would like to understand under what circumstances our processing thread might fail.