-
Notifications
You must be signed in to change notification settings - Fork 0
refactor(SQO): Modularise codebase, create a single source of truth for config, add test framework, review all components. #3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ad_and_validate_config
Move _export_bigquery_data_as_csvs_and_return_lists_of_ineligible_and_eligible_indexers to data processor module as export_bigquery_data_as_csvs_and_return_indexer_lists
Move _clean_old_date_directories to data processor module as clean_old_date_directories
Update issuance_data_access_helper.py Update ruff_check_format_assets.sh update ruff settings Ruff selective turn off ruff ruff linting linting Update custom_formatter.py Update service_quality_oracle.py Update custom_formatter.py Update retry_decorator.py Update custom_formatter.py Update retry_decorator.py
e59bae4
to
7f3a592
Compare
ruff Update configuration.py Fix CI error
2303363
to
617a008
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This pull request reworks the Service Quality Oracle, refactoring it into a fully modular, and more easily testable system. Key improvements include a new modular architecture, centralised configuration management, standardised retry logic, and updated CI/CD workflows.
Key Changes:
1. Modular Architecture:
The application's core logic, previously handled by a few tightly-coupled scripts (including
issuance_eligibility_oracle_core.py
andissuance_data_access_helper.py
), has been refactored into distinct modules undersrc/models
. Each module has a specific responsibility, making the system easier to maintain and test:bigquery_data_access_provider.py
: Manages all BigQuery interactions.blockchain_client.py
: Handles blockchain operations with RPC failover.eligibility_pipeline.py
: Contains the core eligibility logic.subgraph_data_access_provider.py
: Interfaces with the subgraph.scheduler.py
: Manages the scheduling of the oracle's run.service_quality_oracle.py
: Orchestrates the overall workflow, integrating the other modules.2. Centralised Configuration (
src/utils/configuration.py
):config_loader.py
with the more robustconfiguration.py
module.ConfigLoader
for loading settings fromconfig.toml
and substituting environment variables.CredentialManager
class to handle Google Cloud credentials.ConfigurationError
for missing or invalid values.3. Standardised Retries (
src/utils/retry_decorator.py
):retry_with_backoff
decorator using thetenacity
library to provide a consistent exponential backoff strategy.SlackNotifier
has been updated to use this new decorator, simplifying its implementation.4. Build and Dependency Updates:
Dockerfile
anddocker-compose.yml
have been updated to support the new application structure.pyproject.toml
updated.scripts/custom_formatter.py
and integrated into the linting scriptscripts/ruff_check_format_assets.sh
.config.toml.example
has been updated to reflect the new configuration.5. Utility and Code Quality Improvements:
key_validator.py
: Refactoredvalidate_and_format_private_key
to be more concise. It now raises aKeyValidationError
on failure instead of returning a result object.slack_notifier.py
: Simplified by removing manual retry logic in favor of the new decorator.6. Structured Testing Framework (
tests/
):test_blockchain_client.py
test_configuration.py
test_eligibility_pipeline.py
test_scheduler.py
test_service_quality_oracle.py
7. CI/CD and Documentation:
ci.yml
,tests.yml
) have been updated to align with the new project structure and testing framework.README.md
has been updated to reflect the new architecture and provide clear setup and usage instructions.