Task to finish build due inactivity has an edge case

[humitos]

Yesterday while deploying the community site we stop the celery queues for a while and we accumulate many tasks on it. I triggered a couple of builds and they weren't executed by any worker in a period of 1080 seconds (900s + 20%). Because of this reason, these builds were taken by finish_inactive_builds periodic task:

https://github.com/rtfd/readthedocs.org/blob/1f8351b593394efc9f20533776f4e48a018bbb56/readthedocs/projects/tasks.py#L1195-L1206

This task, marked them as FAILED and added the error message to it. After a while I saw my build like this:

Then, when the workers recovered and started executing pending builds from the queue, these builds were performed and now the same build id shows a completely different message: COMPLETED

One possible way to solve this is to save the task_id into the Build object and before marking the build as FAILED we could check through celery if that task_id is still alive or pending for a worker to take it. As far as I know, at this point we have no way to link a Build object with a Celery task.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stsewd]

Is there a way to replicate this locally?

[humitos]

@stsewd yes,

1. you have to start your Django instance with CELERY_ALWAYS_EAGER=False
2. do not run the celery process
3. trigger a build
4. wait for DOCKER_TIME_LIMIT * 1.2

and you will hit the edge case.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stsewd]

Still valid, but very low priority

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dojutsu-user]

Still valid bot.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dojutsu-user]

Still valid.

[humitos]

This function should check if we have a celery task queued and mark the build as finished if not. Otherwise, we may be set as finished builds that are being re-tried because of concurrency. Also, using the queue we will be 100% accurate in our decision and it does not require to be based on "time since the build was triggered" or similar.