checkpoint support

[yogeshVU]

Version information

Observed behavior

If the federate/federation crashes, there needs a way to restore the faulty/crashed federate to the last known stage

Expected behavior

Federate crashes, the federation restores from the earlier known snapshot

[yogeshVU]

Currently, the way I am thinking is:

Send a pause command to the fedmgr to pause the simulation
Checkpoint the whole federation
Restart the simulation docker from the checkpoint
Send a RESUME command to the fedmgr to resume the simulation

[yogeshVU]

Himanshu's reply:

Below are some ideas that might work:

• Let's say you create a checkpointing_period variable (e.g., 10 seconds), then at every 10 seconds of logical time simulation, the federation should be checkpointed.

• For other federates to know about the checkpointing_period, one way is for the federation manager to send an interaction to all federates at the start. Another way is to read it from the configuration files, just like other parameters.

• At the start of the simulation, the federation manager registers a "unique" checkpoint say: <federation_name>checkpoint

• This should be achieved at logical time = 10 seconds.

• All federates simulate up to 10 seconds (even if this timepoint lies within the current step and not at the end of the step) and immediately call readyToCheckpoint() method, which says that it has achieved this checkpoint and is waiting on others to also achieve this checkpoint.

• Once all federates have achieved the checkpoint, the federation manager at the last achieves the checkpoint as well. In its achieveCheckpoint() method, it first:
  -- checkpoints all docker containers for all federates in the federation
  -- registers the next checkpoint with ID <federation_name>checkpoint
  -- calls readyToCheckpoint() method for achieving the UUID1 checkpoint
  -- removes the achieved UUID1 checkpoint from its internal list
  Note that we never PAUSE or RESUME the running federation in this way.

[hneema08]

Ziqi: Thanks for your advice. Still, I have doubts. What do you think about the federation manager itself? Are we supposed to also checkpoint the container of federation manager when the system achieves the checkpoint?

Himanshu's reply: Yes, absolutely. A federation manager is also part of the federation and has "internal state" that maintains and manages the federation.