Game snapshots/save/load for quick catch-up or replay seek.

[badosu]

What are the challenges for enabling the ability to:

• Load a game directly into the most recent game state saved instead of replaying the whole simulation until the current game time.
• Be able to seek into replay timestamps instead of having to simulate and only move forwards in time.

[badosu]

See also https://recoilengine.org/articles/netcode-overview/#replays-and-saves

Just to initiate the discussion:

• Make save/load reliable
• Make save be able to be done asynchronously (or unrealistically fast enough synchronously), so every time you want to perform a save the frame won't lag
• For games: this implies making infra changes to distribute, download and restore saves. This also implies using dedicated instead of headless servers: since you need the state on the server now, unless you want to trust/use one of the players sym and bandwidth to perform the role.
• For replays: this implies some contract for attaching save states to replays (e.g. when loading a replay what are the timestamps I can seek to?)

[lostsquirrel1]

This is an important discussion to have. A topic that comes up often and is worth capturing the findings in an easily referenceable location.

So the address the big point: saves are not sync safe

They only capture a small fraction of the game state, and the game state is effectively reloaded upon loading. This means that a game that runs from frame 0 to frame X and then saved is a different game to one loading straight into frame X from that very same save.

This is a huge task to address, making the save/load game logic work in a sync safe fashion. It would require deep understanding of all areas of the engine's simulation code, completely mastery of deterministic simulation coding and a monumental amount of time, patience, and determination. Most of time will be spent chasing multiple cases of desyncs. Desyncs are some of worst kind of bugs to have to face and are bad enough when there's just one to deal with, but with this work you're very likely going to be getting multiple other desyncs potentially triggering while you're trying fix one.

It isn't impossible, but I could easily see an full-on expert (with no knowledge gaps) take months on a task like this. The rest of us would be lucky if the work is stable enough to consider using within a year.

This is something that should have been developed along-side the rest of the engine if that was the objective. Not likely to have happened, especially when we consider most of our engine updates have broken the save/load system (and had to be fixed.) Keeping the save/load system working as it currently is would be a cake walk in comparison to a full syncing save/load system and we seem to have enough issues with the easier scenario as it is.

[ScarylePoo]

ZWZSG had a pretty full featured lua save/load system. It worked really well.

Editing: Ofc it had all the pitfalls that would come with a lua save/load. Obviously animations are reset, any projectiles that are in the air are no longer there, etc. But, it was a very very good approximation.

[sprunk]

Lua saveload wouldn't help for catchup or replay seek because it would desync.

[lostsquirrel1]

There are also other issues: one of the clients would have to periodically upload a game state to the server. Then you have deal with problems like what happens if the connection is lost mid update, and data management on the server. Also connect rates are very restricted so to prevent any one client from causing a DoS attack either by accident or by purpose - which has an impact on snapshots because a sync-safe game state, even compressed, could be over 10Mb (or possibly even 100mb) by the late game. It could take several minutes upload, which means it would take a similar amount of time for a reconnecting client to download the data - then they would have to attempt a catch up from the difference between the time of the downloaded game state and the current state of the game. So, if for example, it takes 5 minutes to upload, and is done once every ten minutes, then a reconnecting client would have to catch up anywhere from 10-20 in-game minutes, after having to wait 5 minutes to download - and the player has nothing much to look at during the download time, unlike the current reconnect where they get to see the battle play out quickly.

[Hoeze]

I disagree:

• a snapshot can be made on demand, e.g. when another player tries to reconnect
• uploading 100mb takes maybe 10s on a modern connection, even on a slow 10Mbit/s connection it would be only ~80s

Yesterday we had to wait >30min for one of us catching up at 91:55min in the game. Not fun.

[lostsquirrel1]

As for replays, taking an uncompressed sync-safe snapshot is going to require hundreds of megabytes (maybe over a Gigabyte? Who knows?), so you are not going to be able to store many of these. The situation is worse late game because 1) more data is needed and 2) it takes longer to simulate the frames between the snapshots. It is also quiet likely that, due to the architecture, that taking a snapshot will cause stutter (a bad experience issue during replays, but far more of an issue if this was to be done in live games for quick catch up purposes) - and partly why dynamic compression/uncompression is going to cause as many troubles as it solves if one was thinking to use that to solve the memory space issue.

[lostsquirrel1]

The bottom line is that this feature wasn't considered right from the beginning and a series of design decisions have been made over the decades, one after another, that have increasingly lead us away from being able to practically take sync-safe snapshots.

[XNTEABDSC]

Wild talk:

Allow players to start a vote for pausing and saving so delay is accepted.

After saving everyone load the saved snapshot, and all new people joined start simulating at the most recent snapshot, so there are probably no desync problem .