Hollow responses are relabelled as truncation and bisected, so one bad response costs up to 15 billed calls (measured 18x token blow-up)

[yotamleo]

Summary

_response_is_hollow() (llm.py) collapses two very different failures into one, and
_call_* then relabels the result as finish_reason="length" so adaptive retry bisects
the chunk:

• truncated — the model ran out of max_completion_tokens mid-JSON. Bisecting is the
  correct recovery: smaller input ⇒ shorter output ⇒ it fits.
• hollow — HTTP 200 with empty/null/whitespace content, or content that parses to zero
  nodes and zero edges (a rate limit, a transport hiccup, a refusal, an agentic prose reply,
  a reasoning-first content block).

For the hollow case bisecting cannot converge: both halves are produced by the same
misbehaving backend and come back hollow too. Every hollow response therefore costs
2**max_retry_depth billed calls instead of one — with max_retry_depth=3, up to 15
calls per chunk, all of which fail.

Five call sites do the relabel: llm.py ~1292 (_call_openai_compat), 1343 (_call_claude),
1606 (_call_claude_cli), 1669 (_call_azure), 1726 (_call_bedrock).

Evidence

Measured on a scheduled nightly run against two corpora (graphify 0.9.4x, claude-cli):

run	input tokens	output tokens	result
2026-08-16 (healthy)	144,540	10,669	completed, ~82 min
2026-08-17 (storm)	2,585,365	234,277	same corpus, same graph size (920 nodes / 854 edges)

~18x the input tokens for an identical graph — very close to the 15x worst case the
depth-3 fan-out predicts. The log is dominated by pairs of:

[graphify] claude-cli returned a hollow response; treating as truncation so adaptive retry can bisect the chunk.
[graphify] chunk of 15 truncated at depth 0, splitting into halves of 7 and 8

The second corpus that night ran 6h44m and was still on chunk 149/314 when the machine
was rebooted — it never emitted a tokens: line, so the spend was invisible until the log
was read by hand.

The codebase already knows this is the failure mode; the comments say so:

• llm.py ~1510 — "…reads as truncation, so the adaptive-retry path bisects the chunk
  indefinitely, never converging and never writing graph.json"
• llm.py ~1557 — "…reads as truncation, and gets bisected without ever converging (claude-cli backend returns an agentic summary instead of JSON on Claude Code 2.1.216 (the llm.py:1376 workaround has regressed) #2076)"
• _bedrock_response_text ~1075 — "…gets reclassified as truncation, and sends the chunk
  into bisection that cannot converge"

#2076, #2287 and #2866 each fixed one backend-specific source of hollow responses. The
conflation itself is still there, so the next new source costs 15x again.

Asks

1. Do not route hollow into the bisect path. Keep _response_is_hollow as a detector,
   but give it its own recovery: retry the same chunk with backoff (2-3 attempts), then
   fail the chunk loudly. Bisect only on a real finish_reason == "length" or a genuine
   context-window rejection. Bisecting is a size remedy; a hollow response is not a size
   problem.
2. A budget, not just a depth. max_retry_depth bounds the tree's depth, but nothing
   bounds total sub-calls, and there is no run-level ceiling. Add a per-chunk sub-call cap
   and an optional global call/token ceiling that aborts the run.
3. Expose the knob. max_retry_depth is a Python-API kwarg only — there is no CLI flag
   and no environment variable, so a graphify extract operator cannot lower it or set it
   to 0 as a mitigation. An env var (e.g. GRAPHIFY_MAX_RETRY_DEPTH) would be enough.
4. Emit tokens: on abnormal termination too. The counters are accumulated in memory
   and printed only on the normal write path, so a run that is killed, crashes, or is
   interrupted leaves no cost record at all. A finally/atexit emission (or a periodic
   running total) would make a storm visible without reading the whole log.

Asks 1-3 are the fix; 4 is what makes the next occurrence detectable.

[safishamsi]

Fixed in v0.9.48: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.48 via #2888 (a hollow response is retried on the same chunk instead of bisected; GRAPHIFY_MAX_RETRY_DEPTH=0 disables it). Closing as resolved.