Retry DataForSEO Google AI Overview requests on transient server errors

[jrhizor]

Problem

The /status page shows 0% for Google AI Overview on DataForSEO. That cell is a 7-day pass rate, so 0% means the google-ai-overview:dataforseo:online target has failed on every scheduled run since it was added in #469 — each one throwing:

DataForSEO API Error: Internal SE Server Error.

Root cause

Not a parsing bug — the error is thrown at the task-status check in runGoogleAiOverview, before any extraction runs. DataForSEO's organic SERP task returns a non-20000 status when its on-demand load_async_ai_overview step fails on their side.

That failure is transient: running the target manually against the same account passed 10/10 times (7 solo + 3 under the same 5-way parallel DataForSEO load the scheduled job uses). The AI Overview path had no retry, so a single blip hard-fails the run and records a fail. Its sibling from the same PR — BrightData's runGoogleAiOverview — already wraps its call in a 3-attempt backoff, which is why BrightData's AI Overview reads healthy while DataForSEO's reads 0%.

Fix

Wrap the organic SERP call in the same 3-attempt linear backoff (1.5s, 3s) the BrightData runner uses. A transient Internal SE Server Error is retried instead of failing the run; a genuinely failing request still throws the same DataForSEO API Error: … after 3 attempts. Retries only fire on a failed attempt (at most 4×/day per target).

Testing

• packages/lib unit tests pass, including two new cases — retry-then-succeed and throw-after-3-failures — using fake timers.
• tsc clean for @workspace/lib.
• Manually verified 10/10 live AI Overview calls succeed against DataForSEO, including under parallel load.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
elmo	Ready	Preview, Comment	Jul 19, 2026 2:35am