fix(redis): use POST body for setCachedJson to avoid ECONNRESET with large payloads

[bighaoZQH]

Summary

Error writing data to Redis

Type of change

• Bug fix
• New feature
• New data source / feed
• New map layer
• Refactor / code cleanup
• Documentation
• CI / Build / Infrastructure

Affected areas

• Map / Globe
• News panels / RSS feeds
• AI Insights / World Brief
• Market Radar / Crypto
• Desktop app (Tauri)
• API endpoints (/api/*)
• Config / Settings
• Other:

Checklist

• Tested on worldmonitor.app variant
• Tested on tech.worldmonitor.app variant (if applicable)
• New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds)
• No API keys or secrets committed
• TypeScript compiles without errors (npm run typecheck)

Screenshots

Problem

setCachedJson currently encodes the value into the URL path:

await fetch(
  `${url}/set/${encodeURIComponent(key)}/${encodeURIComponent(JSON.stringify(value))}/EX/${ttlSeconds}`,
  { method: 'POST', ... }
);

When the payload is large (e.g. news:digest:v1 is ~126KB),
encodeURIComponent inflates the data further, exceeding Node.js's
default HTTP URL length limit (~16KB). This causes ECONNRESET /
EPIPE errors on every write, so the key is never persisted to Redis.

The data falls back to the in-process fallbackDigestCache Map instead,
which means the cache is lost on every container restart.

As shown in the image, I curled a request to the news module, but the logs showed errors related to the Redis link and that the relevant key was missing:

Root Cause

Node.js http.createServer rejects requests with URLs exceeding ~16KB,
and the redis-rest proxy (redis-rest-proxy.mjs) uses Node.js's built-in
HTTP server. Large payloads in the URL path are silently dropped.

Fix

Switch setCachedJson to use a POST request body instead of URL parameters, consistent with the pattern used by other Redis helpers in this file:

const resp = await fetch(`${url}/`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${token}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(['SET', prefixKey(key), JSON.stringify(value), 'EX', String(ttlSeconds)]),
  signal: AbortSignal.timeout(REDIS_PIPELINE_TIMEOUT_MS),
});

As shown in the image, I modified the setCachedJson method in the redis.ts file, then rebuilt the project and repeated the previous steps. The problem is now resolved, and there are no more Redis-related errors in the logs:

Impact

• Fixes news:digest:v1 never being persisted to Redis in self-hosted deployments
• Fixes any other large payload write failures (same root cause)
• No behavior change for small payloads

[vercel]

@bighaoZQH is attempting to deploy a commit to the Elie Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR fixes a real ECONNRESET / EPIPE bug in setCachedJson where large JSON payloads (e.g., the ~126 KB news:digest:v1 key) exceeded Node.js's ~16 KB URL length limit because the value was encodeURIComponent-encoded directly into the URL path. The fix correctly moves the payload into the request body via the /pipeline endpoint, consistent with every other bulk-write helper in the file.

Key findings:

• The root-cause analysis and the chosen approach are both correct and well-reasoned.
• The new call still uses REDIS_OP_TIMEOUT_MS (1,500 ms) rather than REDIS_PIPELINE_TIMEOUT_MS (5,000 ms) used by all other pipeline callers in the same file. For the ~126 KB payloads that motivated this change, a 1,500 ms budget may still cause intermittent timeouts on slower or containerised networks, partially re-introducing the failure this PR is meant to fix.
• Unlike every other pipeline function in the file, the new call does not check resp.ok, so a non-2xx response (rate-limit, proxy error) is silently ignored and won't appear in logs.

Confidence Score: 4/5

Safe to merge after addressing the pipeline timeout — the core fix is correct but the remaining 1,500 ms timeout may still cause failures for large payloads on slower networks.

The approach is correct and well-motivated. A single P1 remains: the REDIS_OP_TIMEOUT_MS (1,500 ms) is inconsistent with every other pipeline call in the file (all use 5,000 ms) and could partially re-introduce timeout failures for the large payloads this PR targets. A one-line fix addresses it. The P2 (missing resp.ok guard) is pre-existing behaviour and non-blocking.

server/_shared/redis.ts — timeout constant on line 94 should be REDIS_PIPELINE_TIMEOUT_MS

Important Files Changed

Filename	Overview
server/_shared/redis.ts	setCachedJson migrated from URL-path encoding to /pipeline body — correct root-cause fix, but still uses the short 1,500 ms op timeout instead of the 5,000 ms pipeline timeout used by every other pipeline call; also no resp.ok guard unlike all peer functions.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant setCachedJson
    participant RedisRestProxy as Redis REST Proxy (/pipeline)

    Note over Caller,RedisRestProxy: Before (ECONNRESET on large payloads)
    Caller->>setCachedJson: key, value (~126 KB), ttlSeconds
    setCachedJson->>RedisRestProxy: POST /set/{encodeURIComponent(key)}/{encodeURIComponent(value)}/EX/{ttl}
    RedisRestProxy-->>setCachedJson: ECONNRESET / EPIPE (URL > 16 KB)

    Note over Caller,RedisRestProxy: After (body-based pipeline)
    Caller->>setCachedJson: key, value (~126 KB), ttlSeconds
    setCachedJson->>RedisRestProxy: POST /pipeline, body: [[SET, prefixedKey, value, EX, ttl]]
    RedisRestProxy-->>setCachedJson: [{result: OK}]
    setCachedJson-->>Caller: void

Loading

_{Reviews (1): Last reviewed commit: "fix(redis): use pipeline body for setCac..." | Re-trigger Greptile}

[greptile-apps]

Wrong timeout for pipeline endpoint

The fix correctly moves large payloads to the request body, but the timeout is still REDIS_OP_TIMEOUT_MS (1,500 ms) — the same budget as the old single-command URL call. Every other pipeline call in this file (getCachedJsonBatch, geoSearchByBox, getHashFieldsBatch, runRedisPipeline) uses REDIS_PIPELINE_TIMEOUT_MS (5,000 ms), which exists precisely because pipelines can be slower, and because large payload transfers take more time over the wire.

For the news:digest:v1 key (~126 KB), transmitting the body to a self-hosted Redis REST proxy within 1,500 ms may still fail on slower or containerised networks, partially re-introducing the timeout failures this PR is meant to fix.

Suggested change

	signal: AbortSignal.timeout(REDIS_OP_TIMEOUT_MS),
	signal: AbortSignal.timeout(REDIS_PIPELINE_TIMEOUT_MS),

[greptile-apps]

HTTP error response not checked

Every other pipeline function in this file (getCachedJsonBatch, geoSearchByBox, getHashFieldsBatch, runRedisPipeline) checks resp.ok and logs or returns early on non-2xx responses. The new implementation silently swallows a non-2xx status (e.g., a 429 rate-limit or 500 from the proxy), so a persistent write failure won't surface in logs.

Consider adding the same guard the other pipeline helpers use:

const resp = await fetch(`${url}/pipeline`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${token}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(pipeline),
  signal: AbortSignal.timeout(REDIS_PIPELINE_TIMEOUT_MS),
});
if (!resp.ok) {
  console.warn(`[redis] setCachedJson HTTP ${resp.status}`);
}

[bighaoZQH]

request review

[SebastienMelki]

@bighaoZQH — good catch and well-documented fix. Root cause analysis is correct and POST body approach is consistent with other pipeline helpers.

One issue to fix: AbortSignal.timeout uses REDIS_OP_TIMEOUT_MS (1,500ms), but all other pipeline calls use REDIS_PIPELINE_TIMEOUT_MS (5,000ms). For the ~126KB payloads motivating this PR, 1.5s may cause intermittent timeouts. Please change to REDIS_PIPELINE_TIMEOUT_MS.

Once updated, this is ready to merge.