Reconfiguring machine metadata might cause rescheduling

[bcwaldon]

1. start a fleet node with metadata "foo=bar"
2. schedule a unit to the cluster with the option MachineMetadata= foo=bar"
3. wait for the unit to start on the fleet node started in step 1
4. reconfigure the fleet node to have metadata "foo=baz" (change config; systemctl kill --signal SIGHUP fleet.service`

At this point, the unit will be unscheduled from the node as the MachineMetadata condition is no longer met. Is this the correct behavior?

[jonboulle]

Yes

[jonboulle]

:-)

[bcwaldon]

@jonboulle that's how I feel, too, but I want to see if anyone else has a strong opinion

[epipho]

I think it is the correct behavior as long as units that were stopped are restarted on a different node if a suitable match can be made.

What about one shot or batch jobs? I know support isn't in yet, but it feels like they should not be affected by a metadata change since they can't be restarted elsewhere.

[bcwaldon]

@epipho If batch jobs or oneshots use MachineMetadata to find a host, they are going to be subject to this same behavior. Typically batch jobs are designed to be idempotent, so they can be restarted halfway through execution without issue.

[epipho]

The only edge case I can think of is for "semi-stateful" apps like game servers where shutting down the service will end the games in that session.

Perhaps it just doesn't make sense to use metadata filtering for that kind of job.

[bcwaldon]

So it seems like we should just make sure the repercussions of reconfiguring metadata are well documented when we tackle #555