fix(fuel): enable week-on-week price diff computation#2622
Conversation
The WoW computation was silently skipped for most countries because the observedAt guard required different source observation dates between consecutive weekly seeder runs. Since the EU XLSX often embeds the same date across weekly publishes, nearly all 27 EU countries got 0 WoW. Also fix prev snapshot rotation: only overwrite :prev when the existing snapshot is 6+ days old, preventing the seeder from clobbering a freshly backfilled prev snapshot.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
The previous shouldRotatePrev logic checked if prev data was "old enough" but this used the data's fetchedAt, not when the key was written. A backfill with an older fetchedAt would be immediately overwritten by the seeder. Simpler and correct: rotate prev only when wowAvailable is true, meaning the prev data was successfully consumed for WoW computation.
Greptile SummaryThis PR fixes two root causes that kept Confidence Score: 5/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Seeder starts\nreads CANONICAL_KEY:prev] --> B{hasPrevData?}
B -- No --> C[wowAvailable = false\nshouldRotatePrev = true]
B -- Yes --> D{prevTooRecent?\nprevAge < 6 days}
D -- Yes\nbackfill guard --> E[wowAvailable = false\nshouldRotatePrev = false]
D -- No\nnormal weekly run --> F[wowAvailable = true\nshouldRotatePrev = true]
F --> G[Compute WoW per country\nguarded by 15% anomaly threshold]
G --> H[Write data to CANONICAL_KEY\nWrite data to :prev]
C --> I[Write data to CANONICAL_KEY\nWrite data to :prev\nseed initial snapshot]
E --> J[Write data to CANONICAL_KEY\nPreserve :prev backfill]
Reviews (1): Last reviewed commit: "fix(fuel): enable week-on-week price dif..." | Re-trigger Greptile |
scripts/seed-fuel-prices.mjs
Outdated
| countryCount: countries.length, | ||
| }; | ||
|
|
||
| const shouldRotatePrev = !hasPrevData || !prevTooRecent; |
There was a problem hiding this comment.
Readability of
shouldRotatePrev expression
The expression !hasPrevData || !prevTooRecent is a double-negation that requires a moment to parse. It is equivalent to !(hasPrevData && prevTooRecent) — i.e., "rotate unless we have fresh prev data." A brief inline comment (or a positive rewrite) would make the intent immediately clear to the next reader.
| const shouldRotatePrev = !hasPrevData || !prevTooRecent; | |
| const shouldRotatePrev = !hasPrevData || !prevTooRecent; // rotate unless prev exists AND is too recent (backfill guard) |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
The data.gov.my API returns both series_type=level (actual prices) and series_type=change_weekly (delta) rows. The change_weekly row has ron95=0 when price is unchanged, which the seeder rejected as "out of range".
Spain was fetched from minetur.gob.es (live station average) AND the EU XLSX (weekly weighted average). Since minetur.gob.es ran first, it won the merge, causing a source mismatch with the :prev snapshot (EU XLSX). This produced a phantom -10% WoW diff. Removing fetchSpain() makes Spain consistent with all other EU countries, sourced entirely from the EU Oil Bulletin XLSX.
The UK source was fetchUK_ModeA (live retailer station scrape, ~2650 stations) which produced a £0.105/L gap vs the gov.uk DESNZ published weekly average. This caused a phantom +7.3% WoW when compared against the DESNZ-sourced prev snapshot. Replace with fetchUK_DESNZ which reads the official weekly road fuel prices CSV from gov.uk. URL is discovered via the Content API since it changes weekly. This gives consistent WoW and is the authoritative UK national average.
Summary
observedAtguard in WoW computation that silently skipped nearly all countries when the EU XLSX source date didn't change between weekly runs:prevsnapshot when existing prev is 6+ days old, preventing fresh backfill data from being overwrittenContext
The fuel prices panel has had
wowAvailable: falsesince launch. Root cause: theobservedAt !== prev.observedAtcheck on lines 656/665 required different source observation dates between runs. The EU Oil Bulletin XLSX (covering 27 countries) often publishes the same observation date across consecutive weekly updates, causing WoW to be skipped for the majority of countries.The
prevAgecheck (6-day minimum) already prevents stale comparisons, and the 15% anomaly threshold catches data bugs, making theobservedAtguard both redundant and harmful.Additionally, the
extraKeysunconditionally overwrote:prevon every run, which meant running a backfill followed by the seeder would immediately clobber the backfill data.Verification
Manually backfilled March 23 EU prices from the EC historical XLSX, ran the fixed seeder: 27/32 countries now show WoW arrows in production.
Test plan
wowAvailable: truein Redis after seeder runwowPctvalues (missing: MY gasoline=0, MX API down, BR/NZ/UK no prev)npm run typecheckpasses