feat(market): add Alpha Vantage as primary market data source (#2055)#2097
Open
feat(market): add Alpha Vantage as primary market data source (#2055)#2097
Conversation
Add fetchAlphaVantageQuotesBatch() and fetchAlphaVantagePhysicalCommodity() to server/worldmonitor/market/v1/_shared.ts. Update three Railway seed scripts to try ALPHA_VANTAGE_API_KEY first: - seed-market-quotes.mjs: AV REALTIME_BULK_QUOTES (up to 100 symbols/call) → Finnhub secondary → Yahoo fallback (Indian NSE + Yahoo-only symbols) - seed-commodity-quotes.mjs: AV physical functions (WTI/BRENT/NATURAL_GAS/ COPPER/ALUMINUM) + AV bulk for ETF proxies → Yahoo fallback for futures (GC=F, SI=F, PL=F, ^VIX etc.) not covered by AV physical endpoints - seed-etf-flows.mjs: AV REALTIME_BULK_QUOTES for all BTC-spot ETFs → Yahoo 5d chart fallback for uncovered tickers Yahoo Finance and Finnhub remain as fallbacks; Railway relay Yahoo proxy route is not yet removed (pending migration validation). Closes #2055
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
…sleep ordering - Detect AV `Information` rate-limit response in all four AV fetch paths (_shared.ts bulk + physical, seed-market-quotes, seed-commodity-quotes, seed-etf-flows) and break/return early instead of silently continuing - Add 500ms inter-batch delay in fetchAlphaVantageQuotesBatch (both _shared.ts and seed-market-quotes) to avoid bursting AV 150 req/min limit - Fix prevClose guard in _shared.ts: `prevClose && Number.isFinite(prevClose)` replaced with `Number.isFinite(prevClose) && prevClose > 0` for consistency with seed scripts - Fix Yahoo sleep-before-covered-check in seed-commodity-quotes and seed-etf-flows: delay now only fires for actual Yahoo requests, not for AV-already-covered symbols at i > 0
koala73
added
High Value
- Extract shared _shared-av.mjs: fetchAvBulkQuotes + fetchAvPhysicalCommodity (was duplicated in 3 seed scripts and _shared.ts) - Add 1-retry with 1s backoff on all AV fetch calls (network transients) - Log count of dropped symbols when rate limit breaks a batch loop - Populate sparkline from daily data array in fetchAvPhysicalCommodity (last 7 closes, oldest→newest; was always []) - Fix ETF seed: avgVolume=0, volumeRatio=0 when AV covers ticker (REALTIME_BULK_QUOTES has no 5-day history; 1/volume was misleading) - Consolidate NSE filter in seed-market-quotes to use YAHOO_ONLY set (was three separate endsWith/startsWith checks)
GC=F (Gold) and SI=F (Silver) are supported AV physical commodity functions per AV docs and were explicitly listed in issue #2055. Both were missing from AV_PHYSICAL_MAP, falling back to Yahoo. PL=F, PA=F, RB=F, HO=F have no AV physical function — Yahoo fallback for those is correct.
seed-gulf-quotes.mjs: - Oil (CL=F, BZ=F): AV physical functions (WTI/BRENT) as primary - Currencies (SAR/AED/QAR/KWD/BHD/OMR): AV FX_DAILY as primary (gives daily close + change% + 7-point sparkline) - Indices (^TASI.SR, DFMGI.AE, etc.): Yahoo fallback (no AV equivalent) seed-fx-rates.mjs: - AV CURRENCY_EXCHANGE_RATE as primary for all ~50 currencies - 900ms between calls (67 req/min, safe under 75/min limit) - Yahoo fallback for any currencies AV does not cover _shared-av.mjs: - Add fetchAvCurrencyRate() — CURRENCY_EXCHANGE_RATE, single pair → rate - Add fetchAvFxDaily() — FX_DAILY, daily close + change% + sparkline
AV CURRENCY_EXCHANGE_RATE is one-call-per-pair with no bulk equivalent, offers worse exotic-currency coverage than Yahoo (LBP, NGN, KES, VND), and a daily cron doesn't face Yahoo's IP rate-limit pressure. seed-gulf-quotes currencies (6 pairs, 10-min cron) keep AV FX_DAILY.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fetchAlphaVantageQuotesBatch()(up to 100 symbols/call) andfetchAlphaVantagePhysicalCommodity()toserver/worldmonitor/market/v1/_shared.tsseed-market-quotes.mjs:ALPHA_VANTAGE_API_KEY→ AVREALTIME_BULK_QUOTESas primary; Finnhub secondary for non-AV stocks; Yahoo fallback for Indian NSE symbols and any remainingseed-commodity-quotes.mjs: AV physical functions (WTI,BRENT,NATURAL_GAS,COPPER,ALUMINUM) for supported futures; AV bulk quotes for ETF proxies (URA,LIT); Yahoo fallback for unsupported futures (GC=F,SI=F,^VIX, etc.)seed-etf-flows.mjs: AVREALTIME_BULK_QUOTESfor all BTC-spot ETF tickers; Yahoo 5d chart fallback for any not returned by AVWhat changes at runtime
ALPHA_VANTAGE_API_KEYis set in Railway env: AV is used as primary, eliminating Yahoo IP-based 429s and Finnhub burst limitsRequired env var
ALPHA_VANTAGE_API_KEY— add to Railway service environment variables (obtain from https://www.alphavantage.co/premium/)Test plan
tsc --noEmit -p tsconfig.api.json)npm run test:data)ALPHA_VANTAGE_API_KEYin Railway staging, run seed scripts manually, verify Redis keys populatedmarket:commodities-bootstrap:v1has WTI, BRENT, etc. with AV-sourced prices