Add resilient database initialization with retries and improved startup

[vincentmakes]

Summary

This PR improves the reliability of database initialization by adding exponential backoff retry logic and better separation of critical vs. optional migrations. It also enhances the Docker entrypoint to support multi-database scenarios and configurable wait timeouts.

Key Changes

Master DB Initialization (app/services/master_db.py)

• Added asyncio import for async sleep functionality
• Implemented exponential backoff retry logic (up to 5 attempts with 2^n second delays) for initial database connectivity verification
• Improved logging to track connection attempts and failures
• Refactored _apply_pending_migrations() to distinguish between:
  • Critical tables (admin_users, admin_sessions): Must succeed or app fails to start
  • Optional migrations (organizations, SSO, tenant columns): Failures are logged but don't prevent startup
• Updated docstrings to document retry behavior and migration strategy

Docker Entrypoint (docker-entrypoint.sh)

• Enhanced wait_for_db() function with:
  • Configurable label parameter for clearer logging (e.g., "tenant DB" vs "master DB")
  • Configurable max attempts via DB_WAIT_ATTEMPTS environment variable (defaults to 30)
• Added support for multi-tenant scenarios:
  • Waits for tenant database first
  • Conditionally waits for master database if MULTI_TENANT=true and MASTER_DB_HOST differs from tenant DB host
• Improved messaging to indicate which database is being waited for and how many attempts remain

Implementation Details

• The retry logic uses exponential backoff (2, 4, 8, 16, 32 seconds) to gracefully handle PostgreSQL containers that are still initializing
• Critical migrations run outside the try-catch block to ensure fatal errors propagate
• Optional migrations are wrapped in try-catch to allow partial schema initialization
• The Docker entrypoint remains non-fatal for database readiness (app starts regardless), but the Python code now has its own retry mechanism for robustness
• Environment variable DB_WAIT_ATTEMPTS allows operators to tune startup behavior for their infrastructure

https://claude.ai/code/session_01KHhXcB7ZB2eojjBd3tYxtB