Gunicorn workers timeout despite extended timeout settings

[andersy005]

the fastAPI application recently started experiencing [CRITICAL] WORKER TIMEOUT errors. despite configuring a 300-second timeout, workers are still being terminated and restarted, causing API instability due to partial database updates (in some cases, only projects are updated, leaving credits and clips empty). this appears to happen during the database update.

current configuration

gunicorn -w $OFFSETS_DB_WEB_CONCURRENCY -t 300 -k uvicorn.workers.UvicornWorker offsets_db_api.main:app --config gunicorn_config.py --access-logfile '-' --error-logfile '-'

even with the 300-second timeout, the logs indicate workers are timing out and restarting:

[2025-04-10 15:07:54 +0000] [655] [CRITICAL] WORKER TIMEOUT (pid:686)
[2025-04-10 15:07:54 +0000] [655] [ERROR] Worker (pid:686) was sent SIGABRT!
[2025-04-10 15:07:54 +0000] [810] [INFO] Booting worker with pid: 810

[andersy005]

found this issue which might be relevant:

• Workers go into restarting/crash cycle (WORKER TIMEOUT / signal 6) tiangolo/uvicorn-gunicorn-fastapi-docker#145

[badgley]

#166 relates as well.

curious to see more about how switching to multi on inserts affected performance. It looks like multi still uses a SQL INSERT statement. if we need more / better performance, we can try a COPY statement. we also discussed the possibility of completely recreating the database.

In a world where we're going to keep the same database around and instead use this of drop-and-interest pattern, we probably also want to run VACUUM FULL as part of the ETL process. I imagine Postgres keeps track of older rows that we drop — we should clean those out in the event they affect performance over the long run.