feat(e3): story persistence tracking

[koala73]

Summary

• Proto: StoryMeta message + StoryPhase enum added to NewsItem (fields 9-11). importanceScore and corroborationCount stubs added for E1.
• list-feed-digest.ts: Builds a corroboration map across ALL items before truncation (so cross-category mentions of the same story are visible). Batch-reads existing story:track hashes from Redis and attaches StoryMeta to each proto item. Writes HINCRBY/HSET/HSETNX/SADD/EXPIRE per story in 80-story pipeline chunks (~9 commands/story).
• cache-keys.ts: STORY_TRACK_KEY_PREFIX, STORY_SOURCES_KEY_PREFIX, DIGEST_ACCUMULATOR_KEY_PREFIX, STORY_TRACKING_TTL_S (48h).
• src/types/index.ts: StoryMeta, StoryPhase types; NewsItem extended with importanceScore?, corroborationCount?, storyMeta?.
• data-loader.ts: protoItemToNewsItem maps STORY_PHASE_* → client StoryPhase.
• NewsPanel.ts: Phase badges — BREAKING (red), DEVELOPING ×N (orange), ONGOING (muted).

How story phases work

Phase	Condition
BREAKING	First appearance (mentionCount = 1)
DEVELOPING	2-5 mentions, < 2h old
SUSTAINED	6+ mentions or > 2h old and still prominent
FADING	currentScore < peakScore × 0.5 (activates in E1)

Redis keys per story (48h TTL)

story:track:v1:<hash16>    hash  firstSeen, lastSeen, mentionCount, sourceCount, currentScore, peakScore, severity
story:sources:v1:<hash16>  set   feed names (for cross-source corroboration count)

First request writes the keys. Second request (15 min later, after cache miss) reads them back and populates StoryMeta in the response.

Dependency

This is the Redis foundation E1 (Composite Importance Score) needs. E1 will overwrite currentScore/peakScore in the hash and add ZADD GT on story:peak:v1:<hash> when it ships.

Test plan

• Verify typecheck + typecheck:api pass (both clean)
• Verify test:data passes (2748 tests)
• After deploy: confirm story:track:v1:* keys appear in Upstash within 15 min of first digest build
• After two digest cycles (30 min): confirm phase badges appear in NewsPanel

Post-Deploy Monitoring & Validation

• Logs: [list-feed-digest] writeStoryTracking — should be silent on success, warn on Redis failure
• Redis: KEYS story:track:v1:* count should be 200-500 within one digest cycle
• Failure signal: If writeStoryTracking failed appears in logs, story tracking is not writing but digest delivery is unaffected (fire-and-forget error path)
• Rollback: No schema changes, no breaking changes to existing digest response shape. New fields are optional — old clients ignore them.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	Apr 2, 2026 5:17pm

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
WorldMonitor	🟢 Ready	View Preview	Apr 2, 2026, 10:31 AM

[greptile-apps]

Greptile Summary

This PR introduces E3 story persistence tracking: a Redis-backed pipeline that hashes story titles across digest cycles, accumulates mention/source counts, derives lifecycle phases (BREAKING → DEVELOPING → SUSTAINED → FADING), and surfaces them as StoryMeta on NewsItem. The proto, generated stubs, OpenAPI docs, types, data-loader mapping, and NewsPanel badge rendering are all updated consistently.

Two P1 issues need to be resolved before merging:

• normalizeTitle uses [^\w\s] without the u flag, which strips every non-ASCII character. All Arabic (and other non-Latin) story titles normalise to "", causing every such story to share one Redis hash and produce completely wrong mentionCount/storyMeta data.
• The ALERT badge is now gated on !item.storyMeta, so any tracked story with isAlert: true silently loses its alert indicator once it appears in a second digest cycle.

Confidence Score: 4/5

• Two P1 defects need fixes before merging: Unicode hash collision in normalizeTitle and suppressed ALERT badge for tracked stories.
• The Redis pipeline architecture, phase derivation, and proto contract are all sound. However, the Unicode normalisation bug would corrupt story tracking for any non-Latin feed (the system explicitly supports lang: ar), and the ALERT badge regression silently drops threat indicators on recurring high-severity stories — both are current-path failures on the changed code.
• server/worldmonitor/news/v1/list-feed-digest.ts (normalizeTitle regex) and src/components/NewsPanel.ts (ALERT badge guard)

Important Files Changed

Filename	Overview
server/worldmonitor/news/v1/list-feed-digest.ts	Core story tracking logic added: corroboration map, Redis read/write pipeline, phase derivation — two issues: normalizeTitle strips all Unicode (hash collision for non-Latin titles) and derivePhase ordering makes FADING unreachable until E1
src/components/NewsPanel.ts	Phase badges added (BREAKING/DEVELOPING/ONGOING); ALERT badge guard changed to isAlert && !storyMeta, silently suppressing ALERT on any tracked story regardless of threat severity
server/_shared/cache-keys.ts	Four new key prefixes and 48h TTL constant added for E3 story tracking; straightforward, no issues
src/app/data-loader.ts	PROTO_TO_CLIENT_PHASE map and protoItemToNewsItem extension added; mapping is correct, unspecified phase correctly filtered to undefined
src/types/index.ts	StoryPhase, StoryMeta types and optional NewsItem fields added; clean additions, no issues
proto/worldmonitor/news/v1/news_item.proto	StoryMeta message and StoryPhase enum added to NewsItem (fields 9-11); field numbering is correct and backward-compatible

Sequence Diagram

sequenceDiagram
    participant Client
    participant buildDigest
    participant sha256Hex
    participant Redis

    Client->>buildDigest: GET /list-feed-digest

    Note over buildDigest: Fetch & parse all RSS feeds
    buildDigest->>sha256Hex: normalizeTitle(item.title) × N items
    sha256Hex-->>buildDigest: titleHash (16-char hex)
    Note over buildDigest: Build corroborationMap<br/>titleHash → Set<sourceName>

    buildDigest->>Redis: HMGET story:track:v1:<hash> × uniqueHashes (pipeline)
    Redis-->>buildDigest: existing StoryTrack data

    Note over buildDigest: Derive phase per track<br/>Attach StoryMeta to proto items<br/>Truncate to MAX_ITEMS_PER_CATEGORY

    buildDigest-->>Client: ListFeedDigestResponse (with StoryMeta)

    Note over buildDigest: Fire-and-forget write (async)
    buildDigest->>Redis: HINCRBY mentionCount 1<br/>HSET lastSeen/sourceCount/severity<br/>HSETNX firstSeen/currentScore/peakScore<br/>SADD story:sources:v1:<hash><br/>EXPIRE ×2 (48h TTL)<br/>per story in 80-item pipeline chunks

Loading

Comments Outside Diff (3)

1. server/worldmonitor/news/v1/list-feed-digest.ts, line 169-175 (link)

   Unicode titles all collapse to the same hash

   \w in JavaScript (without the u flag) matches only ASCII [A-Za-z0-9_]. Every non-Latin character — Arabic, Chinese, Japanese, Cyrillic, Hebrew — is treated as a "special character" and stripped by [^\w\s]. A title like "الرياض مدينة" normalises to "", and so does every other Arabic-script title. They all produce the same sha256Hex("") key, so every non-Latin story shares one Redis hash: mentionCount, sourceCount, currentScore and peakScore all accumulate against the same entry, and storyMeta returned to clients is completely wrong for all of them.

   The system already accepts lang: 'ar' requests, so this is a current-path failure for any Arabic-language feed. Use a Unicode-aware regex instead:

2. src/components/NewsPanel.ts, line 389 (link)

   ALERT badge silently suppressed for any tracked story

   The guard !item.storyMeta means that once a story has been seen at least twice (i.e. storyMeta is populated), its ALERT tag is never rendered — even when item.isAlert is true. A critical or high-severity threat that recurs across digest cycles will display only a DEVELOPING/ONGOING phase badge, with no indication that it triggered an alert. This is a regression from the previous behaviour where isAlert always rendered the tag.

   The two signals are orthogonal: phase describes lifecycle; isAlert describes threat severity. They should render independently.

3. server/worldmonitor/news/v1/list-feed-digest.ts, line 186-192 (link)

   STORY_PHASE_FADING is unreachable in practice

   derivePhase checks FADING after the SUSTAINED catch-all, so the function always returns SUSTAINED before it can return FADING. The if (currentScore < peakScore * 0.5) branch is also guarded by currentScore > 0 && peakScore > 0, but because HSETNX initialises both fields to '0', this condition is never true until E1 ships and overwrites those placeholder zeros. Until then every story ends up as SUSTAINED.

   Consider moving the FADING check before the SUSTAINED return (or documenting the intentional ordering as an E1 activation detail):

_{Reviews (1): Last reviewed commit: "feat(e3): story persistence tracking" | Re-trigger Greptile}