feat: Implement robust exchange rate fetching and aggregation with multiple providers, caching, and historical storage, along with supporting infrastructure. #111
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR implements a resilient Exchange Rate Service designed to fetch, aggregate, and cache crypto-to-fiat rates. The system utilizes a multi-source strategy (Coinbase, Binance, and CoinGecko) to ensure high availability and data accuracy.
🛠️ Key Features
Multi-Source Aggregation: Fetches data from Coinbase and Binance, using a Weighted Average algorithm to calculate the final rate.
Resilience & Fallbacks: CoinGecko acts as a secondary fallback if primary sources fail.
An API Fallback Chain ensures that a single provider outage doesn't break the service.
Data Quality: Includes Outlier Detection to filter out "flash crash" or "bad data" spikes and a Confidence Scoring system for each rate.
Performance & Caching: * Implemented Redis caching for sub-millisecond lookups.
Integrated Staleness Detection to ensure users never receive expired data.
Infrastructure: * Cron Job: Automated updates every 60 seconds.
Persistence: Historical rate storage for auditing and trend analysis.
📊 Rate Calculation Logic
The service doesn't just pick a number; it follows this flow:
Fetch from all active providers.
Filter out outliers (values that deviate significantly from the median).
Calculate a weighted average based on source reliability.
Apply Spread for business logic.
Cache the result in Redis.
🧪 Testing & Verification
Unit Tests: All API responses are mocked to test edge cases and network timeouts.
Manual Verification: * Verified Cron logs for 60-second intervals.
Simulated API failures by blocking provider endpoints to confirm fallback to CoinGecko.
Checked Redis keys to confirm TTL and sub-millisecond response times.
📝 Definition of Done (Acceptance Criteria Check)
[x] Rates update every 60s.
[x] Graceful handling of source failures.
[x] Alerts triggered for stale rates.
[x] Sub-millisecond lookup via Redis.
Fixes #26