luotsi needs an internal state machine: STANDBY, PROCESSING, ERROR

[elnygren]

Currently Luotsi updates it's checksum state before:
a) rendering the new haproxy config
b) replacing the config at HAPROXY_CONF
c) reloading haproxy

As any any of a,b,c could fail, we have a problem: Luotsi can be in an erroneous state, with haproxy running an older config than what the checksum would make us believe. However, HAPROXY_CONF would always show the correct state, as would be expected.

TODO

• Luotsi should have an internal state machine: STANDBY | PROCESSING | ERROR, where:
  • STANDBY: Luotsi/haproxy is running the most recent settings from Halti master normally
  • PROCESSING: Luotsi has received a new config and is performing steps a,b,c (and possibly SSL related tasks in the future)
  • ERROR: Luotsi PROCESSING failed and the most recent settings from Halti master are NOT being run by haproxy (it is running the previous settings)
• luotsiPoll should set the internal state machine accordingly:
  • PROCESSING when checksum changes and we start processing the new settings
  • STANDBY if processing is successful
  • ERROR if standby is not successful