Offline sender sometimes doesn't send heartbeats

[alahiff]

Here is an example of when the server received heartbeats for a specific run. Note the big gaps in the timestamps:

194.81.223.66 - - [22/Feb/2025:16:35:03 +0000] "PUT /api/runs/V4xXatum2bKBouQdN8JdN5/heartbeat HTTP/1.1" 200 4 "-" "Simvue Python client 2.0.0a1"
194.81.223.66 - - [22/Feb/2025:17:01:03 +0000] "PUT /api/runs/V4xXatum2bKBouQdN8JdN5/heartbeat HTTP/1.1" 200 4 "-" "Simvue Python client 2.0.0a1"
194.81.223.66 - - [22/Feb/2025:17:26:03 +0000] "PUT /api/runs/V4xXatum2bKBouQdN8JdN5/heartbeat HTTP/1.1" 200 4 "-" "Simvue Python client 2.0.0a1"

Sorry, I'm not sure how to reproduce this.

[wk9874]

No idea what could cause this - do you know what the cronjob interval and heartbeat intervals were @alahiff ? And were you running a lot of simulations at once while this was happening?

I've added threading into the heartbeat sender to speed it up a bit in the case of a very large number of runs, but suspect that wont properly fix this issue. If there were a very large number of concurrent runs / uploads / uploading large files that take a long time then the heartbeats will be stuck behind those uploads finishing, so that might slow it down significantly

[alahiff]

Currently being tested.