Skip to content

fix(fuel): enable week-on-week price diff computation#2622

Merged
koala73 merged 5 commits intomainfrom
fix/fuel-wow-computation
Apr 2, 2026
Merged

fix(fuel): enable week-on-week price diff computation#2622
koala73 merged 5 commits intomainfrom
fix/fuel-wow-computation

Conversation

@koala73
Copy link
Copy Markdown
Owner

@koala73 koala73 commented Apr 2, 2026

Summary

  • Remove observedAt guard in WoW computation that silently skipped nearly all countries when the EU XLSX source date didn't change between weekly runs
  • Only rotate current data to :prev snapshot when existing prev is 6+ days old, preventing fresh backfill data from being overwritten

Context

The fuel prices panel has had wowAvailable: false since launch. Root cause: the observedAt !== prev.observedAt check on lines 656/665 required different source observation dates between runs. The EU Oil Bulletin XLSX (covering 27 countries) often publishes the same observation date across consecutive weekly updates, causing WoW to be skipped for the majority of countries.

The prevAge check (6-day minimum) already prevents stale comparisons, and the 15% anomaly threshold catches data bugs, making the observedAt guard both redundant and harmful.

Additionally, the extraKeys unconditionally overwrote :prev on every run, which meant running a backfill followed by the seeder would immediately clobber the backfill data.

Verification

Manually backfilled March 23 EU prices from the EC historical XLSX, ran the fixed seeder: 27/32 countries now show WoW arrows in production.

Test plan

  • Verified wowAvailable: true in Redis after seeder run
  • 27/32 countries have wowPct values (missing: MY gasoline=0, MX API down, BR/NZ/UK no prev)
  • Anomaly threshold correctly flags MY diesel (-90.94%)
  • npm run typecheck passes

The WoW computation was silently skipped for most countries because the
observedAt guard required different source observation dates between
consecutive weekly seeder runs. Since the EU XLSX often embeds the same
date across weekly publishes, nearly all 27 EU countries got 0 WoW.

Also fix prev snapshot rotation: only overwrite :prev when the existing
snapshot is 6+ days old, preventing the seeder from clobbering a freshly
backfilled prev snapshot.
@vercel
Copy link
Copy Markdown

vercel bot commented Apr 2, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
worldmonitor Ignored Ignored Preview Apr 2, 2026 4:05pm

Request Review

The previous shouldRotatePrev logic checked if prev data was "old enough"
but this used the data's fetchedAt, not when the key was written. A backfill
with an older fetchedAt would be immediately overwritten by the seeder.

Simpler and correct: rotate prev only when wowAvailable is true, meaning
the prev data was successfully consumed for WoW computation.
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 2, 2026

Greptile Summary

This PR fixes two root causes that kept wowAvailable: false since launch: (1) the observedAt !== prev.observedAt guard silently skipped WoW for the 27 EU countries because the EC Oil Bulletin XLSX reuses the same observation date across consecutive weekly publishes, and (2) extraKeys unconditionally overwrote :prev on every run, clobbering any manually backfilled historical snapshot. The new shouldRotatePrev flag (!hasPrevData || !prevTooRecent) correctly skips the rotation only when a fresh prev snapshot already exists (< 6 days old), preserving backfill data for the next scheduled run. Logic has been verified in production with 27/32 countries now showing WoW arrows.

Confidence Score: 5/5

  • Safe to merge — both changes are logically correct and production-verified.
  • All remaining findings are P2 (readability). The core logic of removing the observedAt guard and the shouldRotatePrev gate is sound: the three state cases (no prev, prev too recent, prev old enough) are all handled correctly, and the 6-day minimum age + 15% anomaly threshold provide adequate data-quality protection in place of the removed guard.
  • No files require special attention.

Important Files Changed

Filename Overview
scripts/seed-fuel-prices.mjs Removes the over-restrictive observedAt guard on WoW computation and adds a shouldRotatePrev gate to prevent fresh backfill snapshots from being overwritten; logic is correct and verified in production.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Seeder starts\nreads CANONICAL_KEY:prev] --> B{hasPrevData?}
    B -- No --> C[wowAvailable = false\nshouldRotatePrev = true]
    B -- Yes --> D{prevTooRecent?\nprevAge < 6 days}
    D -- Yes\nbackfill guard --> E[wowAvailable = false\nshouldRotatePrev = false]
    D -- No\nnormal weekly run --> F[wowAvailable = true\nshouldRotatePrev = true]
    F --> G[Compute WoW per country\nguarded by 15% anomaly threshold]
    G --> H[Write data to CANONICAL_KEY\nWrite data to :prev]
    C --> I[Write data to CANONICAL_KEY\nWrite data to :prev\nseed initial snapshot]
    E --> J[Write data to CANONICAL_KEY\nPreserve :prev backfill]
Loading

Reviews (1): Last reviewed commit: "fix(fuel): enable week-on-week price dif..." | Re-trigger Greptile

countryCount: countries.length,
};

const shouldRotatePrev = !hasPrevData || !prevTooRecent;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Readability of shouldRotatePrev expression

The expression !hasPrevData || !prevTooRecent is a double-negation that requires a moment to parse. It is equivalent to !(hasPrevData && prevTooRecent) — i.e., "rotate unless we have fresh prev data." A brief inline comment (or a positive rewrite) would make the intent immediately clear to the next reader.

Suggested change
const shouldRotatePrev = !hasPrevData || !prevTooRecent;
const shouldRotatePrev = !hasPrevData || !prevTooRecent; // rotate unless prev exists AND is too recent (backfill guard)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

koala73 added 3 commits April 2, 2026 19:39
The data.gov.my API returns both series_type=level (actual prices) and
series_type=change_weekly (delta) rows. The change_weekly row has ron95=0
when price is unchanged, which the seeder rejected as "out of range".
Spain was fetched from minetur.gob.es (live station average) AND the EU
XLSX (weekly weighted average). Since minetur.gob.es ran first, it won
the merge, causing a source mismatch with the :prev snapshot (EU XLSX).
This produced a phantom -10% WoW diff.

Removing fetchSpain() makes Spain consistent with all other EU countries,
sourced entirely from the EU Oil Bulletin XLSX.
The UK source was fetchUK_ModeA (live retailer station scrape, ~2650
stations) which produced a £0.105/L gap vs the gov.uk DESNZ published
weekly average. This caused a phantom +7.3% WoW when compared against
the DESNZ-sourced prev snapshot.

Replace with fetchUK_DESNZ which reads the official weekly road fuel
prices CSV from gov.uk. URL is discovered via the Content API since it
changes weekly. This gives consistent WoW and is the authoritative
UK national average.
@koala73 koala73 merged commit c897953 into main Apr 2, 2026
7 checks passed
@koala73 koala73 deleted the fix/fuel-wow-computation branch April 2, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant