[Spike] Test performance implications of connection pool

[exalate-issue-sync]

With the initial setup of the connection pool, we need to investigate what it’s impact on performance

Run locust test against load-fecfile-web-api with connection pooling on and off (using the previous CONN_MAX_AGE solution) (IE, before #1887 was merged)

compare results to draw conclusions

test with prod config - large db, 4 instances, 50 users?

QA Notes

null

DEV Notes

null

Design

null

See full ticket and images here: FECFILE-2864

[exalate-issue-sync]

Laura Beaufort commented: {noformat}cf run-task load-fecfile-web-api --command "python django-backend/manage.py gen_locust_load_test_data --number_of_committees=1000 --number_of_reports=8 --number_of_contacts=800 --number_of_receipts=1600 --number_of_disbursements=400 --number_of_debts=8 --number_of_loans=8" --name "load-1000-sm-committees-lb"
cf logs load-fecfile-web-api | grep load-1000-sm-committees-lb

cf run-task load-fecfile-web-api --command "python django-backend/manage.py gen_locust_load_test_data --number_of_committees=500 --number_of_reports=8 --number_of_contacts=800 --number_of_receipts=4800 --number_of_disbursements=1200 --number_of_debts=8 --number_of_loans=8" --name "load-500-lg-committees-lb"
cf logs load-fecfile-web-api | grep load-500-lg-committees-lb{noformat}

[exalate-issue-sync]

Laura Beaufort commented: # Recommend taking a pg_dump of the loaded data in case you want a fresh start without waiting several hours

Figure out starting point for {{DB_POOL_MIN_SIZE}} and {{DB_POOL_MAX_SIZE}} for prod (large DB, 4-6 API and Celery instances - plan for 6 API/Celery instances, run 4 API/Celery instances)

Scale up load mirror site to large db and 4 application instances (API/Celery) (reference manifests and/or [https://docs.google.com/spreadsheets/d/1EsY3zOFv2mv9-f-sf64xbPI8dUGwEawQN2l68aHtO5w/edit?gid=1045134148#gid=1045134148|https://docs.google.com/spreadsheets/d/1EsY3zOFv2mv9-f-sf64xbPI8dUGwEawQN2l68aHtO5w/edit?gid=1045134148#gid=1045134148|smart-link] which should be up to date) - can temporarily scale down production if needed {{cf update-service load-fecfile-api-rds -p large-gp-psql-redundant}}

Run locust with pooling (already in develop) instructions: [https://github.com/fecgov/fecfile-web-api/blob/develop/performance-testing/README.md|https://github.com/fecgov/fecfile-web-api/blob/develop/performance-testing/README.md|smart-link]

Run locust without pooling (manual deploy to mirror - revert PR?) [https://github.com/fecgov/fecfile-web-api/compare/develop...revert-1887-feature/connection-pool?expand=1|https://github.com/fecgov/fecfile-web-api/compare/develop...revert-1887-feature/connection-pool?expand=1|smart-link]