Skip to content

refactor(SQO): Modularise codebase, create a single source of truth for config, add test framework, review all components. #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 48 commits into from
Jun 12, 2025

Conversation

MoonBoi9001
Copy link
Member

@MoonBoi9001 MoonBoi9001 commented Jun 10, 2025

Description:

This pull request reworks the Service Quality Oracle, refactoring it into a fully modular, and more easily testable system. Key improvements include a new modular architecture, centralised configuration management, standardised retry logic, and updated CI/CD workflows.

Key Changes:

1. Modular Architecture:
The application's core logic, previously handled by a few tightly-coupled scripts (including issuance_eligibility_oracle_core.py and issuance_data_access_helper.py), has been refactored into distinct modules under src/models. Each module has a specific responsibility, making the system easier to maintain and test:

  • bigquery_data_access_provider.py: Manages all BigQuery interactions.
  • blockchain_client.py: Handles blockchain operations with RPC failover.
  • eligibility_pipeline.py: Contains the core eligibility logic.
  • subgraph_data_access_provider.py: Interfaces with the subgraph.
  • scheduler.py: Manages the scheduling of the oracle's run.
  • service_quality_oracle.py: Orchestrates the overall workflow, integrating the other modules.

2. Centralised Configuration (src/utils/configuration.py):

  • Replaced config_loader.py with the more robust configuration.py module.
  • Introduced ConfigLoader for loading settings from config.toml and substituting environment variables.
  • Added a CredentialManager class to handle Google Cloud credentials.
  • Implemented validation for configuration fields, raising ConfigurationError for missing or invalid values.

3. Standardised Retries (src/utils/retry_decorator.py):

  • Created a new, reusable retry_with_backoff decorator using the tenacity library to provide a consistent exponential backoff strategy.
  • The SlackNotifier has been updated to use this new decorator, simplifying its implementation.

4. Build and Dependency Updates:

  • Docker: Dockerfile and docker-compose.yml have been updated to support the new application structure.
  • Dependencies: pyproject.toml updated.
  • Formatting: A new custom formatter was introduced in scripts/custom_formatter.py and integrated into the linting script scripts/ruff_check_format_assets.sh.
  • Configuration: config.toml.example has been updated to reflect the new configuration.

5. Utility and Code Quality Improvements:

  • key_validator.py: Refactored validate_and_format_private_key to be more concise. It now raises a KeyValidationError on failure instead of returning a result object.
  • slack_notifier.py: Simplified by removing manual retry logic in favor of the new decorator.
  • Error Handling: Implemented consistent error handling across modules.

6. Structured Testing Framework (tests/):

  • Established a formal testing suite by adding new test files for key components:
    • test_blockchain_client.py
    • test_configuration.py
    • test_eligibility_pipeline.py
    • test_scheduler.py
    • test_service_quality_oracle.py

7. CI/CD and Documentation:

  • The GitHub Actions workflows (ci.yml, tests.yml) have been updated to align with the new project structure and testing framework.
  • The README.md has been updated to reflect the new architecture and provide clear setup and usage instructions.

Move _export_bigquery_data_as_csvs_and_return_lists_of_ineligible_and_eligible_indexers to data processor module as export_bigquery_data_as_csvs_and_return_indexer_lists
Move _clean_old_date_directories to data processor module as clean_old_date_directories
Update issuance_data_access_helper.py

Update ruff_check_format_assets.sh

update ruff settings

Ruff

selective turn off ruff

ruff linting

linting

Update custom_formatter.py

Update service_quality_oracle.py

Update custom_formatter.py

Update retry_decorator.py

Update custom_formatter.py

Update retry_decorator.py
@MoonBoi9001 MoonBoi9001 force-pushed the TODO_CONTINUE_3 branch 6 times, most recently from e59bae4 to 7f3a592 Compare June 11, 2025 11:47
@MoonBoi9001 MoonBoi9001 changed the title merge changes refactor(SQO): Modularise codebase, create a single source of truth for config, add test framework, review all components. Jun 12, 2025
@MoonBoi9001 MoonBoi9001 marked this pull request as ready for review June 12, 2025 12:53
@MoonBoi9001 MoonBoi9001 merged commit 2845036 into main Jun 12, 2025
9 checks passed
@MoonBoi9001 MoonBoi9001 deleted the TODO_CONTINUE_3 branch June 12, 2025 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant