feat(observability): export flow/graph runs to Langfuse with Agent-Graph-view metadata#4499
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 15 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (12)
Comment |
…gfuse export - tinyagents feat/langfuse-langgraph-metadata (d6e98e8): langgraph_node/ langgraph_step Agent-Graph-view keys on graph spans + host span-metadata injector on GraphLangfuseExporter - tinyflows feat/graph-event-journal (eab7cf7): journaled run/resume_with_checkpointer entry points recording GraphObservations into a host-injected GraphEventJournal and surfacing the tinyagents run id on JournaledRunOutcome Claude-Session: https://claude.ai/code/session_01VSwRwLDBXg3o6timchvKn8
…ew metadata
flows_run/flows_resume now attach a per-run in-memory GraphEventJournal
via tinyflows' journaled entry points, and after the run row is
finalized best-effort export the run's durable GraphObservations to
Langfuse through the backend proxy (/telemetry/langfuse/ingestion):
- new src/openhuman/tinyflows/langfuse_export.rs:
export_flow_run_trace builds one trace per run (trace_id + sessionId =
thread_id 'flow:{id}:{uuid}', name 'flow.run:{name}') with a span-
metadata injector stamping flow_id on every span; node spans carry
langgraph_node/langgraph_step so Langfuse renders the Agent Graph view
- run-type tags on the trace-create: 'run:flow' plus a trigger tag
(trigger:rpc | trigger:schedule | trigger:app_event | trigger:resume),
threaded from every flows_run callsite via new FlowRunTrigger enum;
also mirrored into trace metadata (run_type/trigger)
- app version (CARGO_PKG_VERSION) stamped as Langfuse release +
app_version trace metadata
- transport mirrors the agent-turn exporter: backend origin from
effective_backend_api_url, live session bearer, 10s cap, 207
tolerated, all failures logged (grep: '[flows]') and swallowed;
gated on observability.share_usage_data
- tests: trace-config/batch construction (tags, langgraph keys,
flow_id injection, app version), share_usage_data + empty-journal
no-op paths
Claude-Session: https://claude.ai/code/session_01VSwRwLDBXg3o6timchvKn8
…inyagents#11, tinyflows#1) Claude-Session: https://claude.ai/code/session_01VSwRwLDBXg3o6timchvKn8
5814715 to
fb91eae
Compare
|
Vendor pins updated to the upstream merge commits (tinyagents 2f023e8 = #11, tinyflows 438f8fc = #1); branch rebased onto latest main (post-#4498). One compat fix folded into the pin bump: upstream tinyagents #8 added optional |
docs/whatsapp-data-flow.md never existed; point the WhatsApp Agent Retrieval row at the domain README instead. Pre-existing on main, surfaced by the Markdown Link Check lane on this PR. Claude-Session: https://claude.ai/code/session_01VSwRwLDBXg3o6timchvKn8
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb91eae8e2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| flow_id, | ||
| thread_id, | ||
| status, | ||
| FlowRunTrigger::Resume, |
There was a problem hiding this comment.
For approval-gated flows, build_flow_trace_config uses the existing thread_id as the Langfuse trace_id, so this resume export writes another trace-create for the same trace. Passing FlowRunTrigger::Resume here re-tags a scheduled/app-event/RPC run as trigger:resume after the user approves it, which corrupts the trigger filters this change adds for completed approval flows. Persist and reuse the original trigger for the run, or use a distinct trace id if resume segments are meant to be separate traces.
Useful? React with 👍 / 👎.
| &journal, | ||
| &journaled.graph_run_ids.run_id, | ||
| ) | ||
| .await; |
There was a problem hiding this comment.
With observability.share_usage_data enabled by default, this awaited Langfuse export runs before flows_run notifies pending approvals or returns the RPC result. If the backend proxy is slow or unreachable, a completed or paused flow can sit here until the 10s export timeout; trigger-driven runs also keep the in-flight guard held during that wait, so scheduled/app-event ticks for the same flow are skipped because telemetry is slow. Since the export is best-effort, clone the observations and run it in the background, or at least notify/return before awaiting it.
Useful? React with 👍 / 👎.
1 participant
What
Flow (workflow-graph) runs are invisible in Langfuse today:
tinyflowsdrives the compiled tinyagents state graph without attaching a journal, so noGraphObservations are ever captured, and the vendoredGraphLangfuseExporteris dead code. This PR wires the whole path:New
src/openhuman/tinyflows/langfuse_export.rs—export_flow_run_trace(...): trace_id +sessionId= flow thread id, nameflow.run:{flow_name}, metadata{flow_id, status, source: flows, run_type: flow, trigger, app_version},release=CARGO_PKG_VERSION, per-spanflow_idvia the exporter's new metadata injector. Sends through the existing backend proxy (/telemetry/langfuse/ingestion, session bearer) — clients still never hold Langfuse keys. Best-effort (warn-and-swallow), gated onobservability.share_usage_data.flows/ops.rs—flows_run/flows_resumenow use tinyflows' journaled entry points (in-memory journal per run) and export afterfinish_flow_run_row.Run-type tags:
run:flow+trigger:rpc|schedule|app_event|resumevia a newFlowRunTriggerenum threaded through every callsite (RPC, cron schedule tick, Composio app event, resume).Vendor submodule bumps (the reason this is a draft): depends on
langgraph_node/langgraph_stepspan metadata (lights up Langfuse's Agent Graph view) + host span-metadata injector;The submodule gitlinks currently point at the fork-branch commits; once the two vendor PRs merge I will bump the pins to the upstream merge commits and mark ready for review. CI cannot fetch the pins until then.
Known limitation
Errored/timed-out flow runs return
Errwithout run ids, so they are not exported (journal slice unreadable without the id) — surfacing ids on the error path needs a tinyflows error-type change, flagged as follow-up in tinyflows#1. Completed and pending-approval runs (including handled node failures, which appear as ERROR spans) export fine.Testing
GGML_NATIVE=OFF cargo checkclean;flows::276 passed,tinyflows::19 passed--no-verify: pre-push hook fails on pre-existing TS breakage onorigin/main(@xyflow/reactunresolved), unrelated to this Rust-only diff.Part of the Langfuse observability overhaul: tinyhumansai/backend#1069, openhuman#4498, follow-up issue openhuman#4496 (scores).
https://claude.ai/code/session_01VSwRwLDBXg3o6timchvKn8