fix(observability): populate Langfuse generation/tool I/O and trace metadata (#6)

[senamakel]

Summary

Fixes #6 — Langfuse generations and tool calls rendered with a name and timing
but empty Input/Output, and harness traces had empty metadata. The event
model was payload-free by design, so the exporter had nothing to forward.

This carries model/tool payloads through the event model behind an opt-in,
default-off capture policy (privacy-preserving by default), populates the
exporter's input/output body fields from them, and defaults useful
trace-level metadata on the harness path.

Changes

• PayloadCapture policy (src/harness/runtime/types.rs): new
  RunPolicy.capture field, PayloadCapture { model_io, tool_io }, defaulting
  to payload-free. PayloadCapture::all() opts in.
• Event model (src/harness/events/types.rs): optional input/output
  (serde default, skip-if-none, backward compatible) on ModelCompleted and
  ToolCompleted.
• Agent loop (src/harness/agent_loop/mod.rs): when capture is enabled,
  snapshots the request messages + completion, and tool args + result, onto the
  completion events. Snapshots happen before the value moves into the model/tool
  wrap onion, so execution is unperturbed. Captured payloads still flow through
  RedactingSink.
• Harness Langfuse exporter (src/harness/observability/langfuse.rs):
  populates generation/tool input/output from the event fields; defaults
  trace metadata (root/first run ids + parent) so a trace is correlatable even
  with no caller metadata, merging caller keys on top.
• Tests: exporter unit tests asserting body.input/body.output for a
  captured generation and tool call and caller-metadata merge; a new e2e test in
  tests/e2e_observability.rs that drives a real harness run with capture on and
  asserts the exported batch carries the I/O and defaulted metadata.
• Docs: new "Payload Capture" section in
  docs/modules/harness/observability.md.

Design notes

• Default stays payload-free. Privacy/PII-sensitive deployments are
  unaffected unless they opt in; the fields are absent from serialized events
  when disabled.
• Graph path. Graph node/step spans are structural and carry no model/tool
  I/O — the real generations and tool calls are the child agent runs, exported
  through the harness client and unified under the same traceId (root run id).
  So the graph exporter needs no I/O change for Langfuse integration: events tracked but bodies (input/output) and trace metadata are empty #6; it already folds
  graph/health into trace metadata.
• Proxy. No backend proxy change required — once the SDK includes
  input/output in the batch they flow through, as the issue notes.

Commands run locally

• cargo fmt --check
• cargo clippy --all-targets -- -D warnings
• cargo test (all suites pass, incl. new exporter + e2e tests)

https://claude.ai/code/session_019zY42oUyeXGxHGesQkTKWV

Summary by CodeRabbit

• New Features

  • Added optional payload capture for runs, allowing model requests/responses and tool inputs/outputs to be included in completion events when enabled.
  • Exported observability data can now include richer trace metadata and captured payload details for downstream tools.

• Bug Fixes

  • Preserved payload-free behavior by default, so existing event streams remain unchanged unless capture is turned on.
  • Improved replay consistency so captured payloads are retained in durable run history and exported views.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds an opt-in PayloadCapture policy to RunPolicy, extends AgentEvent::ModelCompleted/ToolCompleted with optional input/output fields, snapshots model/tool payloads in the agent loop when enabled, and exports them plus default trace metadata via the Langfuse exporter. Documentation and tests are updated accordingly.

Changes

Payload capture feature

Layer / File(s)	Summary
PayloadCapture policy and RunPolicy wiring src/harness/runtime/types.rs	Adds PayloadCapture struct (model_io, tool_io, all(), is_disabled()) and wires a capture field into RunPolicy, defaulting to payload-free.
AgentEvent schema extension src/harness/events/types.rs	Extends ModelCompleted and ToolCompleted with optional, serde-skipped input/output JSON fields.
Agent loop payload snapshotting src/harness/agent_loop/mod.rs	Snapshots request/response and tool arguments/results into captured input/output values gated by capture flags, and includes them when emitting completion events.
Langfuse export of captured payloads and trace metadata src/harness/observability/langfuse.rs	Adds trace_metadata helper merging default lineage with caller metadata, computes trace-level metadata in build_ingestion_batch, and serializes input/output into generation/tool-create bodies with updated unit tests.
Documentation and supporting test fixture updates docs/modules/harness/observability.md, tests/e2e_observability.rs, tests/e2e_orchestrator_subagents.rs, tests/e2e_registry_observability_contracts.rs, src/harness/observability/test.rs, src/harness/testkit/*	Documents payload capture behavior, adds an e2e test exercising captured payloads through Langfuse, and updates existing fixtures/docstrings for the new event fields.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AgentHarness
  participant Model
  participant Tool
  participant LangfuseClient

  AgentHarness->>AgentHarness: policy.capture.model_io?
  AgentHarness->>Model: send request.messages
  Model-->>AgentHarness: response.message
  AgentHarness->>AgentHarness: emit ModelCompleted(input, output, usage)

  AgentHarness->>AgentHarness: policy.capture.tool_io?
  AgentHarness->>Tool: call.arguments
  Tool-->>AgentHarness: tool result
  AgentHarness->>AgentHarness: emit ToolCompleted(input, output)

  AgentHarness->>LangfuseClient: build_ingestion_batch(observations)
  LangfuseClient->>LangfuseClient: trace_metadata(default lineage + caller metadata)
  LangfuseClient-->>LangfuseClient: generation-create/tool-create with input/output/metadata

Loading

Possibly related PRs

• tinyhumansai/tinyagents#3: Modifies the same src/harness/observability/langfuse.rs event-to-batch mapping logic that this PR extends with input/output/metadata fields.

Poem

A rabbit peeked inside the trace,
"No input here, no output's place!"
Now hop by hop, the payloads flow,
Into Langfuse, so we know
What the model said, what tools have done —
Captured snapshots, opt-in fun! 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding payload capture for Langfuse I/O and trace metadata.
Linked Issues check	✅ Passed	The changes add opt-in model/tool payload capture, forward it to Langfuse, and default trace metadata as requested in #6.
Out of Scope Changes check	✅ Passed	The docs and test updates support the observability fix and there is no clear unrelated code in the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0149256710

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep captured payloads out of Langfuse metadata

When PayloadCapture::model_io or tool_io is enabled, the same captured input/output values added here are also serialized into body.metadata.event because the metadata object above embeds obs.event wholesale. For captured Langfuse exports this stores prompts, completions, tool args, and tool results in metadata in addition to the dedicated Input/Output fields, which defeats consumers that hide or scrub only observation I/O and unnecessarily duplicates sensitive payloads. Strip the capture fields from the metadata copy, or avoid embedding the full event there, before exporting captured runs.

Useful? React with 👍 / 👎.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/harness/observability/langfuse.rs (1)

308-358: 🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

Captured input/output get duplicated into metadata.event.

The per-observation metadata object embeds the whole obs.event (Line 315), which now — for ModelCompleted/ToolCompleted under PayloadCapture::all() — includes the same input/output values already placed at body.input/body.output (Lines 334-335, 354-355). Every captured generation/tool observation thus ships the full prompt/completion/tool payload twice, needlessly inflating the Langfuse ingestion body (and risking size-limit issues for large contexts).

♻️ Proposed fix: stop re-embedding the full event in metadata

 fn observation_event(trace_id: &str, obs: &AgentObservation) -> Value {
     let timestamp = iso_ms(obs.ts_ms);
     let metadata = json!({
         "run_id": obs.run_id.as_str(),
         "root_run_id": obs.root_run_id.as_str(),
         "parent_run_id": obs.parent_run_id.as_ref().map(|id| id.as_str()),
         "offset": obs.offset,
-        "event": obs.event,
+        "event_kind": obs.event.kind(),
     });

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/harness/observability/langfuse.rs` around lines 308 - 358, The Langfuse
observation payload is duplicating captured input/output because
`observation_event` puts the full `obs.event` inside `metadata.event` while
`ModelCompleted` and `ToolCompleted` already include the same values in
`body.input` and `body.output`. Update `observation_event` to stop re-embedding
the full event in `metadata` and keep only lightweight fields there, so the
generated `generation-create` and `tool-create` bodies don’t carry duplicate
payloads.

🧹 Nitpick comments (1)

src/harness/agent_loop/mod.rs (1)

447-486: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Full-transcript snapshot on every model call — storage growth caveat.

captured_input serializes request.messages, which is the entire accumulated transcript at that point in the run (built via ModelRequest::new(messages.clone()), where messages is pushed to after every model/tool turn). With model_io capture enabled, each ModelCompleted event therefore stores the whole prior transcript again, so per-run journaled payload size grows roughly quadratically with turn count for long-running conversations. This mirrors how other LLM observability tools render a generation's "Input" (the full prompt sent), so it's likely intentional, but it's worth calling out explicitly as a cost/storage tradeoff in the docs (docs/modules/harness/observability.md) so operators enabling PayloadCapture::model_io on long-running agents understand the storage implications.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/harness/agent_loop/mod.rs` around lines 447 - 486, The `captured_input`
logic in `src/harness/agent_loop/mod.rs` intentionally snapshots the full
`request.messages` transcript for each `AgentEvent::ModelCompleted`, but this
can cause quadratic journal growth when `PayloadCapture::model_io` is enabled.
Keep the behavior as-is in `run_wrapped_model`/`emit`, and update the
observability docs (`docs/modules/harness/observability.md`) to clearly warn
that `model_io` captures the entire prompt transcript on every model call and
can significantly increase storage for long-running conversations.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/harness/observability/langfuse.rs`:
- Around line 308-358: The Langfuse observation payload is duplicating captured
input/output because `observation_event` puts the full `obs.event` inside
`metadata.event` while `ModelCompleted` and `ToolCompleted` already include the
same values in `body.input` and `body.output`. Update `observation_event` to
stop re-embedding the full event in `metadata` and keep only lightweight fields
there, so the generated `generation-create` and `tool-create` bodies don’t carry
duplicate payloads.

---

Nitpick comments:
In `@src/harness/agent_loop/mod.rs`:
- Around line 447-486: The `captured_input` logic in
`src/harness/agent_loop/mod.rs` intentionally snapshots the full
`request.messages` transcript for each `AgentEvent::ModelCompleted`, but this
can cause quadratic journal growth when `PayloadCapture::model_io` is enabled.
Keep the behavior as-is in `run_wrapped_model`/`emit`, and update the
observability docs (`docs/modules/harness/observability.md`) to clearly warn
that `model_io` captures the entire prompt transcript on every model call and
can significantly increase storage for long-running conversations.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0df4c864-74c5-40d4-a39c-2ef141946dc0

📥 Commits

Reviewing files that changed from the base of the PR and between a950018 and 0149256.

📒 Files selected for processing (11)

• docs/modules/harness/observability.md
• src/harness/agent_loop/mod.rs
• src/harness/events/types.rs
• src/harness/observability/langfuse.rs
• src/harness/observability/test.rs
• src/harness/runtime/types.rs
• src/harness/testkit/test.rs
• src/harness/testkit/types.rs
• tests/e2e_observability.rs
• tests/e2e_orchestrator_subagents.rs
• tests/e2e_registry_observability_contracts.rs