[Tracker] OpenHuman agent Langfuse test — findings & fixes (Groups 1–8 + EXT, 2026-07)

[M3gA-Mind]

Findings from driving the OpenHuman desktop assistant with Langfuse tracing on (thread-5e39f26f-51fa-4d8d-b813-d6b5a9572aba, staging, 2026-07-07). Each turn = one agent.turn:orchestrator trace on fuse.tinyhumans.ai. Group 1 (single-agent tool use) run of 4 turns.

Trace IDs (evidence):

• store: thread-5e39f26f-51fa-4d8d-b813-d6b5a9572aba:4777e429-1819-4f08-b3b2-b972d1efdad5 — manage_profile_memory 29s, 2 model retries, 57s total
• read-back (in-thread): thread-5e39f26f-51fa-4d8d-b813-d6b5a9572aba:2ce05532-4a35-4993-a22b-fc316e2bbe63 — no tool, 2.6s
• retrieve (Priya Nair): thread-5e39f26f-51fa-4d8d-b813-d6b5a9572aba:b1546030-5ed4-4c82-833a-3f443be6f8bd — retrieve_memory 81s, SUBAGENT_INCOMPLETE
• goals: thread-5e39f26f-51fa-4d8d-b813-d6b5a9572aba:b49a0c0d-a646-4d1f-8b5a-6ac11af2cb22 — goal_get 0.0s, 5.5s total

Resolvable issues (sub-issues filed separately, linked below):

1. Memory retrieval sub-agent exhausts its tool-call budget and fails [SUBAGENT_INCOMPLETE] on empty/degraded memory — no early-exit. (filed)
2. Model name absent on Langfuse generations → cost always $0 — token counts are sent but model=null, so pricing never maps. (filed)
3. Sub-agent internal steps aren't emitted as nested Langfuse traces — retrieve_memory/manage_profile_memory are opaque single spans; the loop that hit the limit is invisible. (filed)

Also observed (lower priority / infra / harness-side):

• Context bloat: ~29–32K input tokens per model call for trivial asks, growing each turn (ties to TinyAgents token/context work in [Tracker] TinyAgents harness improvements — 2026-07 audit #4632).
• Transient model-call retries (2× on the store turn) added latency — backend inference proxy / routing.
• Trace payload explosion: 300–386 observations/turn, ~145 middleware.started+completed pairs each (one turn ≈ 688 KB) — observability noise + ingestion cost.
• Environment note: this instance's embeddings provider is unconfigured (memory_doctor = unhealthy, 10 failed jobs) — semantic recall is off, degrading all memory ops to slow keyword scans. Partly explains the memory latency; issue Feat/gitbooks #1 is that the agent doesn't degrade gracefully.

[M3gA-Mind]

Sub-issues filed:

• [memory] retrieve_memory sub-agent exhausts its tool-call budget → SUBAGENT_INCOMPLETE instead of failing fast on empty/degraded memory #4655 — [memory] retrieve_memory sub-agent exhausts tool-call budget → SUBAGENT_INCOMPLETE (the Feat/gitbooks #1 root cause)
• [observability] Model name not stamped on Langfuse generations → agent-run cost always $0 #4656 — [observability] model name absent → agent-run cost always $0
• [observability] Sub-agent internal steps not traced to Langfuse — delegated runs are opaque single spans #4657 — [observability] sub-agent internal steps not traced (opaque spans)

Remaining tracker items (context bloat, transient retries, trace payload explosion, embeddings-unconfigured degradation) left inline here for now.

[M3gA-Mind]

Status update — full picture across Groups 1–3

Driving the assistant with Langfuse tracing on surfaced 10 issues; 3 fixes are already merged, 3 PRs are in review, 1 is in progress, 1 is routing/ops, and 2 remain open. Every finding is backed by a specific trace/span with IDs on its issue.

✅ Merged

• [observability] Langfuse traces dropped for large turns — ingestion batch >500 events rejected (no chunking) #4675 → PR fix(observability): chunk Langfuse ingestion batches to <=500 events #4676 — Large-turn traces dropped (>500-event ingestion batch rejected). Fixed by chunking; validated live on 1,999- and 2,918-event turns.
• [observability] Sub-agent internal steps not traced to Langfuse — delegated runs are opaque single spans #4657 → PR fix(tracing): link spawned sub-agent Langfuse traces to their parent turn #4678 — Sub-agent traces not linked to parent (null parent/root run id). Fixed.
• [memory] retrieve_memory sub-agent exhausts its tool-call budget → SUBAGENT_INCOMPLETE instead of failing fast on empty/degraded memory #4655 → PR Fail fast in retrieve_memory sub-agent on empty/degraded memory #4658 — retrieve_memory sub-agent exhausted its tool-call budget on empty/degraded memory (returned SUBAGENT_INCOMPLETE). Early-exit added.

🔍 In review (W5 CI + code-review)

• [observability] Model name not stamped on Langfuse generations → agent-run cost always $0 #4656 → tinyagents PR refactor: split tauri host from openhuman_core runtime #43 — Model/provider name absent on generations → cost always $0. (Root cause was the tinyagents crate exporter, not openhuman.)
• [agent][perf] Independent memory retrievals run sequentially (~141s that should be ~40s) #4680 → PR orchestrator: batch independent memory lookups into one parallel fan-out #4683 — Independent memory retrievals serialized (~141s). Prompt-level fix to batch them; LGTM, awaiting merge.
• [agent] Orchestrator uses async spawn for a review-before-finalize task — critique runs after finalize #4681 → PR fix(orchestrator): route review-before-finalize tasks to synchronous delegation #4682 — Orchestrator used async spawn for a "critique-before-finalize" task, so the critique ran ~10 min after finalize. Sync-delegation fix.

🔨 In progress

• [agent][perf] spawn_parallel_agents runs workers sequentially + truncates each at a 120s cap #4685 — spawn_parallel_agents runs its workers sequentially (zero overlap) and truncates each at a hard 120s cap. Group-3 "run researchers in parallel" ran 3× 120s serial (~360s). Fix in progress.

📋 Open (not yet worked)

• [memory][perf] retrieve_memory takes ~30–40s per call even with data — dominates agent-turn latency #4677 — retrieve_memory takes ~30–40s even with data (4× in one turn = ~141s). Happy-path memory latency.

↪️ Routing / ops (closed — no code fix)

• [agent][perf][routing] Agent turns get 0 cached tokens — route multi-call tiers to cache-returning providers (not a code fix) #4679 — Agent turns get 0 cached tokens (~217K input/turn uncached). Investigation showed the caching machinery is already complete; the lever is routing multi-call tiers to cache-returning providers (DeepSeek/Qwen) away from GMI-Kimi/General-Compute. No code PR warranted → closed as a routing recommendation.

Cross-cutting observations (context, not separately filed)

• Every trace is ~99% low-value events (per-token model.delta + per-middleware start/complete) — this is why turns exceed the 500-event cap; reducing it would fix the batch problem at source (a Langfuse/observability concern, deprioritized vs. app efficiency).
• ~30K input tokens re-sent per model call (7+ calls/turn); compounds with the caching gap ([agent][perf][routing] Agent turns get 0 cached tokens — route multi-call tiers to cache-returning providers (not a code fix) #4679).

Superseded: the original tracker body above covered only the first Group-1 pass.

[M3gA-Mind]

Update (2026-07-10) — Full-suite eval complete (Groups 1–8 + EXT), 16 findings, fixes dispatched

Extended the original Group 1–3 run to the entire prompt suite using a headless scripted harness (drives channel_web_chat via the core socket, records the full nested agent event stream to JSONL — richer than the Langfuse view for timing/spans). 33 traces across Groups 2–8 + live EXT probes on staging, core healthy throughout.

5 root causes (ranked)

• A. Sub-agent reduce-hang (dominant, HIGH): parent turn never folds a sub-agent's output back → empty reply / stalls on heartbeats. Breaks nearly every delegated prompt. Concrete lead: a sub-agent composio_execute tool_call arrived with malformed args (stray ] + literal <tool_call|> template marker) — an unresolvable tool_call that hangs the parent.
• B. No parallel fan-out (HIGH, the key test): 0 parallel spawns across the whole suite — "in parallel"/"council" prompts either single-spawn or serialize 145–200s apart; even spawn_async_subagent serializes.
• C. Base inference stream stall (HIGH, independent of delegation): long single-agent reasoning streams freeze mid-generation and the upstream drops with no terminal event (no chat_done/chat_error).
• D. Over-delegation / mis-routing (MED): trivial local queries, plans, memory recalls, and pure writing tasks spawn sub-agents instead of a direct tool/answer; plan tasks also trip a plan_review_request gate that parks headless.
• E. Turn interference (MED, global): a still-in-flight turn cancels the NEXT turn (even on a different thread) at t=0 ("Cancelled by newer request").
• Secondary: composio_connect (act path) hangs vs graceful read path; ~183k input-token bloat on trivial delegated ops (~38% cached); manage_profile_memory returns 0-char output (persistence unconfirmed); managed-tier cost telemetry reports $0.

Findings → issues → PRs

Root	Issue	PR
A (mechanism: malformed tool-args)	#4766	tinyagents #45
A (graceful-degrade symptom)	#4746	#4751
B (parallel fan-out)	#4754	#4757
C (stream stall / keepalive)	#4761	#4763
D (over-delegation)	#4744	#4752
D-adjacent (fresh-thread continuity)	#4745	#4748
E (turn interference)	#4760	#4765
Memory hang + 0-char write	#4762	#4764
composio_connect fail-fast	#4756	#4758
Token bloat / context trimming	#4755	(in progress)

All PRs are being driven to green + code-reviews-addressed + merge-ready (never auto-merged). Agent-harness root causes (A/B/C often) route to tinyagents per the harness-first policy.

What works (bounds the problem)

Fast non-delegated path is healthy ("Ok." → 5s); delegation completes when the sub-agent finishes in 1–2 fast calls (file ops 108s, live Calendar 73s). Calendar + Gmail integrations are LIVE. So A/B are about multi-call/slow/multi-spawn work, not a total inability to delegate.

Not testable in this harness (need a supervised-tier core)

Approval gates (#19/#20), readonly (#21), no-progress circuit breaker (#23), cancellation (#33).

Note on the original observability findings

The three original sub-issues (memory-retrieval budget, model=null/$0 cost, sub-agent nested traces) plus the headless-Langfuse-ingestion gap (#4739) are observability-side and lower priority — the current effort focuses on agent/app efficiency per direction, not Langfuse polish. The 0-char manage_profile_memory write and cost=$0 telemetry are re-confirmed here.