Fix token counters: post-finish usage frame, rate-limit wipe, session fidelity, estimate disclosure

[davidrhodus]

The report: "token counters sometimes stop counting"

Wire-level diagnosis (HI_DEBUG_STREAM against pipenetwork / glm-5.2-fast):

[2] finish_reason='stop'  usage=no
[3] finish_reason=None    usage=YES {"prompt_tokens":0,"completion_tokens":0,...}  choices=0

With stream_options.include_usage (hi's default), spec-compliant providers send usage in a separate chunk after the finish chunk. The collector broke on finish_reason — chunk [3] was never read. So streamed calls got zero provider usage and every counter has been running on chars/4 estimates via backfill_missing_usage. On pipenetwork specifically the frame is currently all zeros anyway (so numbers there remain estimates even post-fix) — but the frame is now captured wherever it's populated, and labeled honestly either way.

Three more defects found on the same trail:

Defect	Symptom
Usage::add set rate_limits = other.rate_limits unconditionally	any zero-usage booking (side-call, error, estimate) wiped the last observed rate-limit state — display blanks mid-session
backfill_missing_usage never set context_occupancy	occupancy stayed 0 on the estimate path — consumers of the occupancy gauge blinded
Session JSONL persisted only input/output	resume silently dropped cache counters from session totals

Fixes

• Bounded post-finish drain — after finish_reason, keep reading for the trailing usage frame under a 2s POST_FINISH_USAGE_GRACE, so a provider that holds the stream open after finish still can't hang a completed answer (the guarantee break-on-finish gave). Usage attached to the finish chunk itself still terminates immediately — zero added latency on the normal path.
• Usage::add: rate-limits only replaced by a booking that actually carries a snapshot ("latest observed" semantics).
• Usage.estimated (new, serde-default): set whenever a token field was backfilled from a chars/4 estimate; sticky across add(); surfaced as usage_estimated in --report. Estimated numbers can no longer masquerade as provider-metered ones — on pipenetwork today, usage_estimated: true.
• backfill_missing_usage also backfills context_occupancy.
• Session fidelity: usage record now persists cache_read/cache_creation/estimated (serde defaults — old session files still load); resume restores full totals.

Tests

• usage_in_post_finish_chunk_is_captured — the headline regression (spec chunk shape)
• all_zero_usage_frame_marks_output_estimate — the exact observed pipenetwork shape
• add_preserves_last_observed_rate_limits_and_sticks_estimated
• usage_round_trips_cache_tokens_and_estimated_marker (+ legacy-line compat already covered by load_history_skips_corrupted_lines)
• Full workspace suite green (hi-ai 111, hi-agent 389, all others); fmt + clippy -D warnings clean.

Live verification

Identical bare call, old binary vs fixed: same totals (1809/1/1810), 1.2s end-to-end (drain exits on the usage frame — no grace penalty), and the report now carries usage_estimated: true on pipenetwork.

Caveat worth knowing: pipenetwork's usage frame is all zeros (their side), so counters there remain hi-side estimates — now disclosed. The frame does carry ipop.cost_usd; capturing provider-billed cost is a possible follow-up.

🤖 Generated with Claude Code