Fail gracefully

[Veldhoen]

Currently, the ASR task in Airflow fails every so often when the Whisper service gets evicted or OOMKilled. The triggerer will inquire about the status of a task ID that does not exist in the newly started Whisper service pod.
It would be nice if we can prevent this somehow, so that the same resources doesn't have to be processed for ASR again.
We should think about a good way to go about this. One proposal that we might discuss:

On receiving sigterm, stay alive until the task has finished AND Airflow trigger has been informed (that is: the API has sent a Status.DONE: status.HTTP_200_OK response upon a @api.get("/tasks/{task_id}") request.

[greenw0lf]

@Veldhoen should the proposed solution be implemented? Or is this still up for debate?

[Veldhoen]

I think it's open for debate so if you have ideas about it: shoot :)

[gb-beng]

Related: #139

[greenw0lf]

Starting point: https://stackoverflow.com/questions/71528875/signal-handling-in-uvicorn-with-fastapi

[greenw0lf]

UPDATE: The shutdown event for FastAPI blocks all incoming requests, therefore we cannot wait for a get request for the current task to be received before shutting down the app. So, the alternative, for now, is to let the current task finish, then program something in the Airflow triggerer that checks in S3 whether a specific program has been transcribed already or not.

[greenw0lf]

This was also discussed with Sara and, for now, we don't see solutions for this issue that could be implemented in the worker and not in Airflow