diff --git a/AGENTS.md b/AGENTS.md index 93d815d5..bed009fe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,6 @@ Skills use YAML frontmatter with `allowed-tools` — this is required by Cursor ### Setup Skills | Skill | Description | |-------|-------------| -| `sentry-setup-ai-monitoring` | Instrument OpenAI/Anthropic/Vercel AI/LangChain/Google GenAI | | `sentry-otel-exporter-setup` | Setup OTel Collector with Sentry Exporter | | `sentry-span-streaming-js` | Migrate JavaScript SDK from transaction-based to streamed span delivery | | `sentry-span-streaming-python` | Migrate Python SDK from transaction-based to streamed span delivery | diff --git a/README.md b/README.md index 0dfd4254..e512a1eb 100644 --- a/README.md +++ b/README.md @@ -111,7 +111,6 @@ To build any target locally, run `src/plugins//build.sh ` | Skill | What it does | |-------|--------------| -| `sentry-setup-ai-monitoring` | Instrument OpenAI, Anthropic, LangChain, Vercel AI, Google GenAI | | `sentry-otel-exporter-setup` | Configure OTel Collector with the Sentry exporter for multi-project routing | | `sentry-create-alert` | Create alerts via the Sentry workflow engine API | | `sentry-snapshots-cocoa` | Set up Sentry Snapshots for Apple/Cocoa projects | diff --git a/src/SKILL_TREE.md b/src/SKILL_TREE.md index 92552de5..da8039ea 100644 --- a/src/SKILL_TREE.md +++ b/src/SKILL_TREE.md @@ -49,5 +49,4 @@ Configure specific Sentry capabilities beyond basic SDK setup. |---|---| | Create Sentry alerts using the workflow engine API | [`sentry-create-alert`](skills/sentry-create-alert/SKILL.md) | | Configure the OpenTelemetry Collector with Sentry Exporter for multi-project routing and automatic project creation | [`sentry-otel-exporter-setup`](skills/sentry-otel-exporter-setup/SKILL.md) | -| Setup Sentry AI Agent Monitoring in any project | [`sentry-setup-ai-monitoring`](skills/sentry-setup-ai-monitoring/SKILL.md) | | Full Sentry Snapshots setup for Apple/Cocoa projects | [`sentry-snapshots-cocoa`](skills/sentry-snapshots-cocoa/SKILL.md) | diff --git a/src/skills/sentry-feature-setup/SKILL.md b/src/skills/sentry-feature-setup/SKILL.md index f724e76b..3c8b2bb1 100644 --- a/src/skills/sentry-feature-setup/SKILL.md +++ b/src/skills/sentry-feature-setup/SKILL.md @@ -1,6 +1,6 @@ --- name: sentry-feature-setup -description: Configure specific Sentry features beyond basic SDK setup. Use when asked to monitor AI/LLM calls, set up OpenTelemetry pipelines, create alerts and notifications, or set up Sentry Snapshots. +description: Configure specific Sentry features beyond basic SDK setup. Use when asked to set up OpenTelemetry pipelines, create alerts and notifications, or set up Sentry Snapshots. license: Apache-2.0 role: router --- @@ -9,20 +9,19 @@ role: router # Sentry Feature Setup -Configure specific Sentry capabilities beyond basic SDK setup — AI monitoring, OpenTelemetry pipelines, alerts, and Sentry Snapshots. This page helps you find the right feature skill for your task. +Configure specific Sentry capabilities beyond basic SDK setup — OpenTelemetry pipelines, alerts, and Sentry Snapshots. This page helps you find the right feature skill for your task. ## Start Here — Read This Before Doing Anything **Do not skip this section.** Do not assume which feature the user needs. Ask first. -1. If the user mentions **AI monitoring, LLM tracing, conversations, or instrumenting an AI SDK** (OpenAI, Anthropic, LangChain, Vercel AI, Google GenAI, Pydantic AI) → `sentry-setup-ai-monitoring` -2. If the user mentions **OpenTelemetry, OTel Collector, or multi-service telemetry routing** → `sentry-otel-exporter-setup` -3. If the user mentions **alerts, notifications, on-call, Slack/PagerDuty/Discord integration, or workflow rules** → `sentry-create-alert` -4. If the user mentions **Apple/Cocoa snapshot testing or Sentry Snapshots for Apple platforms** — SnapshotPreviews, Apple Snapshots, Cocoa snapshots, Xcode snapshot testing, Swift previews for Sentry Snapshots, iOS, macOS, tvOS, watchOS, or visionOS → `sentry-snapshots-cocoa`. +1. If the user mentions **OpenTelemetry, OTel Collector, or multi-service telemetry routing** → `sentry-otel-exporter-setup` +2. If the user mentions **alerts, notifications, on-call, Slack/PagerDuty/Discord integration, or workflow rules** → `sentry-create-alert` +3. If the user mentions **Apple/Cocoa snapshot testing or Sentry Snapshots for Apple platforms** — SnapshotPreviews, Apple Snapshots, Cocoa snapshots, Xcode snapshot testing, Swift previews for Sentry Snapshots, iOS, macOS, tvOS, watchOS, or visionOS → `sentry-snapshots-cocoa`. When unclear, **ask the user** which feature they want to configure. Do not guess. -> Instrumenting a signal (tracing/spans, logging, metrics, span streaming, or deciding what to emit) is now part of the standalone `sentry-instrument` skill. +> Instrumenting a signal — tracing/spans, logging, metrics, AI/LLM monitoring, or deciding what to emit — is part of the standalone `sentry-instrument` skill. --- @@ -30,7 +29,6 @@ When unclear, **ask the user** which feature they want to configure. Do not gues | Feature | Skill | |---|---| -| AI/LLM monitoring and conversations — instrument OpenAI, Anthropic, LangChain, Vercel AI, Google GenAI, Pydantic AI | [`sentry-setup-ai-monitoring`](../sentry-setup-ai-monitoring/SKILL.md) | | Sentry Snapshots for Apple/Cocoa — upload Apple snapshot images to Sentry; prefer SnapshotPreviews when Swift previews exist | [`sentry-snapshots-cocoa`](../sentry-snapshots-cocoa/SKILL.md) | | OpenTelemetry Collector with Sentry Exporter — multi-project routing, automatic project creation | [`sentry-otel-exporter-setup`](../sentry-otel-exporter-setup/SKILL.md) | | Alerts via workflow engine API — email, Slack, PagerDuty, Discord | [`sentry-create-alert`](../sentry-create-alert/SKILL.md) | diff --git a/src/skills/sentry-setup-ai-monitoring/SKILL.md b/src/skills/sentry-setup-ai-monitoring/SKILL.md deleted file mode 100644 index 36f62c4e..00000000 --- a/src/skills/sentry-setup-ai-monitoring/SKILL.md +++ /dev/null @@ -1,435 +0,0 @@ ---- -name: sentry-setup-ai-monitoring -description: Setup Sentry AI Agent Monitoring in any project. Use when asked to monitor LLM calls, track AI agents, track conversations, or instrument OpenAI/Anthropic/Vercel AI/LangChain/Google GenAI/Pydantic AI. Detects installed AI SDKs and configures appropriate integrations. -license: Apache-2.0 -category: feature-setup -parent: sentry-feature-setup -disable-model-invocation: true ---- - -> [All Skills](../../SKILL_TREE.md) > [Feature Setup](../sentry-feature-setup/SKILL.md) > AI Monitoring - -# Setup Sentry AI Agent Monitoring - -Configure Sentry to track LLM calls, agent executions, tool usage, and token consumption. - -## Invoke This Skill When - -- User asks to "monitor AI/LLM calls" or "track OpenAI/Anthropic usage" -- User wants "AI observability" or "agent monitoring" -- User asks about token usage, model latency, or AI costs - -**Important:** The SDK versions, API names, and code samples below are examples. Always verify against [docs.sentry.io](https://docs.sentry.io) before implementing, as APIs and minimum versions may have changed. - -## Prerequisites - -AI monitoring requires **tracing enabled** (`tracesSampleRate > 0`). - -If the app has multi-turn chats, set a conversation ID by default anywhere it makes sense to identify a chat session. Sentry uses `gen_ai.conversation.id` to group related AI spans into Conversations. Some integrations infer it automatically, but many setups need to set it explicitly. - -## Data Capture Warning - -**Prompt and output recording captures user content that is likely PII.** In JavaScript, genAI input/output capture is **on by default** (governed by `dataCollection.genAI`); in Python it is enabled via `send_default_pii=True`. Before relying on this capture (or per-integration overrides — `recordInputs`/`recordOutputs` in JS, `include_prompts` in Python), confirm: - -- The application's privacy policy permits capturing user prompts and model responses -- Captured data complies with applicable regulations (GDPR, CCPA, etc.) -- Sentry data retention settings are appropriate for the sensitivity of the data - -**Ask the user** whether they want prompt/output capture enabled. Do not enable prompt/output capture without explicit confirmation. Use `tracesSampleRate: 1.0` only in development; in production, use a lower value or a `tracesSampler` function. - -## Detection First - -**Always detect installed AI SDKs before configuring:** - -```bash -# JavaScript -grep -E '"(openai|@anthropic-ai/sdk|ai|@langchain|@google/genai)"' package.json - -# Python -grep -E '(openai|anthropic|langchain|huggingface)' requirements.txt pyproject.toml 2>/dev/null -``` - -## Sampling Check - -After detecting AI SDKs, check the current sampling configuration: - -```bash -# JavaScript -grep -E 'tracesSampleRate|tracesSampler' sentry.*.config.* instrument.* src/instrument.* app/instrument.* 2>/dev/null - -# Python -grep -E 'traces_sample_rate|traces_sampler' *.py **/*.py 2>/dev/null -``` - -**If `tracesSampleRate` / `traces_sample_rate` is below 1.0 AND no `tracesSampler` / `traces_sampler` is configured:** - -Ask the user: - -> "Your current sample rate is {rate}. Agent runs are sampled as complete span trees — if the root span is dropped, all child gen_ai spans are lost. For full AI visibility, gen_ai-related transactions should be sampled at 100%. Would you like me to set up a `tracesSampler` that keeps AI traces at 100% while sampling other traffic at your current rate?" - -If user confirms, read `${SKILL_ROOT}/references/sampling.md` for implementation patterns. - -## Supported SDKs - -### JavaScript - -| Package | Integration | Min Sentry SDK | Auto? | -|---------|-------------|----------------|-------| -| `openai` | `openAIIntegration()` | 10.53.0 | Yes | -| `@anthropic-ai/sdk` | `anthropicAIIntegration()` | 10.53.0 | Yes | -| `ai` (Vercel) | `vercelAIIntegration()` | 10.53.0 | Yes* | -| `@langchain/*` | `langChainIntegration()` | 10.53.0 | Yes | -| `@langchain/langgraph` | `langGraphIntegration()` | 10.53.0 | Yes | -| `@google/genai` | `googleGenAIIntegration()` | 10.53.0 | Yes | - -*Vercel AI: 10.53.0+ required. Requires `experimental_telemetry` per-call. - -### Python - -Integrations auto-enable when the AI package is installed — no explicit registration needed: - -| Package | Auto? | Notes | -|---------|-------|-------| -| `openai` | Yes | Includes OpenAI Agents SDK | -| `anthropic` | Yes | | -| `langchain` / `langgraph` | Yes | | -| `huggingface_hub` | Yes | | -| `google-genai` | Yes | | -| `pydantic-ai` | Yes | | -| `litellm` | **No** | Requires explicit integration | -| `mcp` (Model Context Protocol) | Yes | | - -## JavaScript Configuration - -### Node.js — auto-enabled integrations - -Just ensure tracing is enabled. Integrations auto-enable when the AI package is installed: - -```javascript -Sentry.init({ - dsn: "YOUR_DSN", - tracesSampleRate: 1.0, // Lower in production (e.g., 0.1) - // OpenAI, Anthropic, Google GenAI, LangChain integrations auto-enable in Node.js -}); -``` - -To customize (e.g., enable prompt capture after user confirmation — see Data Capture Warning): - -```javascript -Sentry.init({ - dsn: "YOUR_DSN", - tracesSampleRate: 1.0, - dataCollection: { - // To disable sending user data and HTTP bodies, uncomment the lines below. For more info visit: - // https://docs.sentry.io/platforms/javascript/configuration/options/#dataCollection - // userInfo: false, - // httpBodies: [], - }, - integrations: [ - Sentry.openAIIntegration({ - // recordInputs/recordOutputs default to true (governed by dataCollection.genAI) - }), - ], -}); -``` - -### Browser / Next.js OpenAI (manual wrapping required) - -In browser-side code or Next.js meta-framework apps, auto-instrumentation is not available. Wrap the client manually: - -```javascript -import OpenAI from "openai"; -import * as Sentry from "@sentry/nextjs"; // or @sentry/react, @sentry/browser - -const openai = Sentry.instrumentOpenAiClient(new OpenAI()); -// Use 'openai' client as normal -``` - -### LangChain / LangGraph (auto-enabled) - -```javascript -Sentry.init({ - dsn: "YOUR_DSN", - tracesSampleRate: 1.0, - dataCollection: { - // To disable sending user data and HTTP bodies, uncomment the lines below. For more info visit: - // https://docs.sentry.io/platforms/javascript/configuration/options/#dataCollection - // userInfo: false, - // httpBodies: [], - }, - integrations: [ - Sentry.langChainIntegration(), - Sentry.langGraphIntegration(), - ], -}); -``` - -### Vercel AI SDK - -Add to `sentry.edge.config.ts` for Edge runtime: -```javascript -Sentry.init({ - dsn: "YOUR_DSN", - tracesSampleRate: 1.0, - dataCollection: { - // To disable sending user data and HTTP bodies, uncomment the lines below. For more info visit: - // https://docs.sentry.io/platforms/javascript/configuration/options/#dataCollection - // userInfo: false, - // httpBodies: [], - }, - integrations: [Sentry.vercelAIIntegration()], -}); -``` - -Enable telemetry per-call: -```javascript -await generateText({ - model: openai("gpt-4o"), - prompt: "Hello", - experimental_telemetry: { - isEnabled: true, - recordInputs: true, - recordOutputs: true, - }, -}); -``` - -## Python Configuration - -Integrations auto-enable — just init with tracing. Only add explicit imports to customize options: - -```python -import sentry_sdk - -sentry_sdk.init( - dsn="YOUR_DSN", - traces_sample_rate=1.0, # Lower in production (e.g., 0.1) - send_default_pii=True, - # Integrations auto-enable when the AI package is installed. - # Only specify explicitly to customize (e.g., include_prompts): - # integrations=[OpenAIIntegration(include_prompts=True)], -) -``` - -## Manual Instrumentation - -Use when no supported SDK is detected. Follow the canonical [Sentry Conventions for `gen_ai.*` attributes](https://getsentry.github.io/sentry-conventions/attributes/gen_ai/) — the [JS docs](https://docs.sentry.io/platforms/javascript/guides/connect/ai-agent-monitoring/#manual-instrumentation) may lag behind; do not set attributes marked deprecated in the conventions. - -### Span Types - -| `op` | Span `name` pattern | Purpose | -|------|---------------------|---------| -| `gen_ai.{operation}` (e.g. `gen_ai.chat`, `gen_ai.request`) | `{operation} {model}` (e.g. `chat gpt-4o`) | Individual LLM call | -| `gen_ai.invoke_agent` | `invoke_agent {agent_name}` | Agent execution lifecycle | -| `gen_ai.execute_tool` | `execute_tool {tool_name}` | Tool/function call | -| `gen_ai.handoff` | `handoff from {source} to {target}` | Agent-to-agent transition | - -For LLM-call spans, the `op` follows the pattern `gen_ai.{gen_ai.operation.name}` — use `gen_ai.chat`, `gen_ai.embeddings`, `gen_ai.generate_content`, or `gen_ai.text_completion` where the operation is known. Span attributes only accept primitives; arrays/objects must be JSON-stringified. - -### Example (JavaScript) - -```javascript -const inputMessages = [ - { role: "user", parts: [{ type: "text", content: "Tell me a joke" }] }, -]; - -await Sentry.startSpan({ - op: "gen_ai.chat", - name: "chat gpt-4o", - attributes: { - "gen_ai.request.model": "gpt-4o", - "gen_ai.operation.name": "chat", - "gen_ai.input.messages": JSON.stringify(inputMessages), - }, -}, async (span) => { - const result = await llmClient.complete(inputMessages); - - const outputMessages = [ - { - role: "assistant", - parts: [ - // Thinking/reasoning content goes in a `reasoning` part, NOT a `text` part. - // Sentry surfaces it separately and filters it out of the Conversations view. - { type: "reasoning", content: result.reasoning }, - { type: "text", content: result.text }, - ], - finish_reason: result.finishReason, - }, - ]; - span.setAttribute("gen_ai.output.messages", JSON.stringify(outputMessages)); - span.setAttribute("gen_ai.usage.input_tokens", result.inputTokens); - span.setAttribute("gen_ai.usage.output_tokens", result.outputTokens); - return result; -}); -``` - -### Key Attributes - -**Common (all AI spans):** - -| Attribute | Required | Description | -|-----------|----------|-------------| -| `gen_ai.request.model` | Yes | Model identifier (e.g., `gpt-4o`, `claude-sonnet-4-6`) | -| `gen_ai.operation.name` | No | Operation label (`chat`, `embeddings`, `invoke_agent`, `execute_tool`, `handoff`, etc.) | -| `gen_ai.agent.name` | No | Agent name (set on agent and tool spans) | - -**Model config (LLM call spans):** - -| Attribute | Description | -|-----------|-------------| -| `gen_ai.request.reasoning_effort` | Reasoning effort level for reasoning models (e.g., `low`, `medium`, `high`). Supported values vary by provider. | - -**Request / response content (PII — enable only after confirming; see Data Capture Warning above):** - -| Attribute | Description | -|-----------|-------------| -| `gen_ai.input.messages` | JSON-stringified array of input messages. Each item uses `{role, parts}` where `parts` is `[{type, content}]`; `role` is `"user"`, `"assistant"`, `"tool"`, or `"system"`. Common part `type`s: `"text"`, `"reasoning"`, `"tool_call"`, `"tool_call_response"` | -| `gen_ai.output.messages` | JSON-stringified array of response messages (text + tool calls), same shape as inputs | - -**Thinking / reasoning messages:** Models with extended thinking (Anthropic `thinking` blocks, Gemini `thought`, DeepSeek `reasoning_content`) produce internal reasoning that isn't part of the user-visible reply. Represent it as a `reasoning` part inside the assistant message — `{"type": "reasoning", "content": "..."}` — alongside the user-facing `text` part. Sentry surfaces reasoning parts separately and filters them out of the user-facing Conversations view, so do **not** fold thinking into a `text` part. When previous thinking is fed back into a multi-turn request, include the same `reasoning` parts in the assistant messages within `gen_ai.input.messages`. Record reasoning token counts via `gen_ai.usage.output_tokens.reasoning` (a subset of `gen_ai.usage.output_tokens`). -| `gen_ai.system_instructions` | System prompt passed to the model | -| `gen_ai.tool.definitions` | JSON-stringified list of tools available to the model | - -**Token usage:** - -| Attribute | Description | -|-----------|-------------| -| `gen_ai.usage.input_tokens` | Total input tokens — **includes** cached tokens | -| `gen_ai.usage.input_tokens.cached` | Subset of input tokens served from cache | -| `gen_ai.usage.input_tokens.cache_write` | Tokens written to cache while processing input | -| `gen_ai.usage.output_tokens` | Total output tokens — **includes** reasoning tokens | -| `gen_ai.usage.output_tokens.reasoning` | Subset of output tokens used for reasoning | -| `gen_ai.usage.total_tokens` | Sum of input + output tokens | - -**Tool spans (`gen_ai.execute_tool`):** - -| Attribute | Description | -|-----------|-------------| -| `gen_ai.tool.name` | Tool identifier | -| `gen_ai.tool.description` | Human-readable tool description | -| `gen_ai.tool.call.arguments` | JSON-stringified tool arguments | -| `gen_ai.tool.call.result` | JSON-stringified tool result | - - -### Token Usage and Cost Calculation - -Sentry uses token attributes to [calculate model costs](https://docs.sentry.io/ai/monitoring/agents/costs/). **Cached and reasoning tokens are subsets, not separate counts** — `gen_ai.usage.input_tokens` already includes `gen_ai.usage.input_tokens.cached`, and `gen_ai.usage.output_tokens` already includes `gen_ai.usage.output_tokens.reasoning`. - -Sentry subtracts the cached/reasoning counts from the totals to compute the uncached/non-reasoning portion. Reporting a cached or reasoning count greater than its total produces negative costs in the dashboard. - -Example — 100 input tokens total, 90 served from cache: - -- Correct: `input_tokens = 100`, `input_tokens.cached = 90` -- Wrong: `input_tokens = 10`, `input_tokens.cached = 90` (cached larger than total → negative cost) - -The same rule applies to `gen_ai.usage.output_tokens` vs. `gen_ai.usage.output_tokens.reasoning`. - -## Verification - -After configuring, make an LLM call and check the Sentry Traces dashboard. AI spans appear with `gen_ai.*` operations showing model, token counts, and latency. - -## Conversations - -Conversations gives a readable, chat-style view of past sessions with your AI agent. It groups spans by `gen_ai.conversation.id` — so whether a user talked across multiple traces or multiple conversations happened inside one trace, you get a timeline of every message, tool call, and response. - -When the user asks for AI monitoring setup, proactively mention this requirement if the app has multi-turn chats. Without a conversation ID, the agent-monitoring spans still work, but the Conversations view cannot group the session correctly. - -Find it at **Explore > Conversations** in Sentry. - -### Prerequisites for Conversations - -- Tracing enabled with `tracesSampleRate > 0` -- Gen AI span streaming is on by default — `streamGenAiSpans` defaults to `true` since JS SDK 10.61.0 and `stream_gen_ai_spans` defaults to `True` since Python SDK 2.64.0. This sends AI spans as standalone items, so spans with large inputs/outputs don't hit transaction payload size limits and get dropped. (The options are available since JS 10.53.0 / Python 2.60.0 if you need to set them explicitly on older SDKs.) -- **Input and output capture enabled** — Conversations reconstructs the chat from `gen_ai.input.messages` and `gen_ai.output.messages` attributes. In JS this is on by default (via `dataCollection`); in Python, set `send_default_pii=True`. Without it, conversations appear empty. - -### Setting a Conversation ID - -Some integrations (OpenAI Agents SDK for Python, OpenAI SDK for Node) infer the conversation ID automatically. For all others, set it manually. - -Use a short, opaque identifier — alphanumeric characters with dashes or underscores only. Never use a URL, email address, or other free-form text as the conversation ID: Sentry uses it as a URL path segment, and a value containing a slash breaks Conversations for that session. - -Good examples: -- A UUID: `48e35936-82ab-4f1a-beaf-b2fa4273ac5e` -- A prefixed ID: `conv_5j66UpCpwteGg4YSxUnt7lPYU`, `asst_abc12345`, `sess_987654` - -#### JavaScript - -```javascript -import * as Sentry from "@sentry/node"; // or @sentry/nextjs, @sentry/nestjs, etc. - -// Set at the start of a conversation -Sentry.setConversationId("conv_abc123"); - -// All subsequent AI calls carry gen_ai.conversation.id: "conv_abc123" -await openai.chat.completions.create({ - model: "gpt-5.5", - messages: [{ role: "user", content: "Hello" }], -}); -``` - -#### Python - -```python -import sentry_sdk.ai - -# Set at the start of a conversation -sentry_sdk.ai.set_conversation_id("conv_abc123") - -# All subsequent AI calls carry gen_ai.conversation.id = "conv_abc123" -``` - -Some integrations infer the conversation ID automatically. For example, the Python OpenAI integration picks it up when you use the `conversation` parameter: - -```python -import openai -import sentry_sdk - -sentry_sdk.init(...) - -conversation = openai.conversations.create() -response = openai.responses.create( - model="gpt-5.4", - input=[{"role": "user", "content": "What are the 5 Ds of dodgeball?"}], - conversation=conversation.id # automatically sets gen_ai.conversation.id -) -``` - -### User Attribution - -The Conversations view shows a **User** column. To populate it, call `setUser` / `set_user` once per request or session, before any AI calls: - -#### JavaScript - -```javascript -import * as Sentry from "@sentry/node"; // or @sentry/nextjs, @sentry/nestjs, etc. - -Sentry.setUser({ id: "user_123", email: "jane@example.com", username: "jane" }); -``` - -#### Python - -```python -import sentry_sdk - -sentry_sdk.set_user({"id": "user_123", "email": "jane@example.com", "username": "jane"}) -``` - -Any of `id`, `email`, or `username` is sufficient — Conversations will display whichever fields are present. - -### Conversations vs Traces - -These are independent concepts: -- A single conversation can span **multiple traces** (e.g., user refreshes the page mid-conversation — new trace, same conversation ID) -- A single trace can contain spans from **different conversations** (e.g., user starts a new chat without refreshing) - -## Troubleshooting - -| Issue | Solution | -|-------|----------| -| AI spans not appearing | Verify `tracesSampleRate > 0`, check SDK version | -| Token counts missing | Some providers don't return tokens for streaming | -| Negative or wrong costs in dashboard | Cached/reasoning tokens are subsets of totals — see Token Usage and Cost Calculation | -| Prompts not captured | In JS, genAI capture is on by default — ensure you haven't set `dataCollection: { genAI: { inputs: false } }`, or pass `recordInputs: true` explicitly. In Python, set `send_default_pii=True`; use `include_prompts` only for explicit overrides | -| Vercel AI not working | Add `experimental_telemetry` to each call | -| Conversations view empty | Ensure Gen AI span streaming is enabled (default since JS SDK 10.61.0 / Python SDK 2.64.0), genAI input/output capture enabled (on by default in JS via `dataCollection`; `send_default_pii=True` in Python), and a conversation ID is set | -| User column shows "Unknown" | Call `Sentry.setUser()` (JS) or `sentry_sdk.set_user()` (Python) once per request or session | diff --git a/src/skills/sentry-setup-ai-monitoring/references/sampling.md b/src/skills/sentry-setup-ai-monitoring/references/sampling.md deleted file mode 100644 index 30aa84cd..00000000 --- a/src/skills/sentry-setup-ai-monitoring/references/sampling.md +++ /dev/null @@ -1,82 +0,0 @@ -# Sampling Strategy for AI Agent Spans - -> `@sentry/node` >=9.x (`inheritOrSampleWith`), `sentry-sdk` >=2.x (`traces_sampler`) - -## The Problem - -Agent runs are span trees. Sampling decides at the root; children inherit. Drop the root, lose every child span. At any rate below 1.0, you lose entire agent executions. - -## How It Works - -`tracesSampler` / `traces_sampler` only fires on **root spans**. Non-root spans (including `gen_ai.*` children) inherit unconditionally. - -**Scenario 1: gen_ai span IS the root** (cron, queue consumer, CLI). The sampler sees `gen_ai.*` directly. Match and return 1.0. - -**Scenario 2: gen_ai spans are children of HTTP transactions** (most web apps). `POST /api/chat` is sampled before any AI code runs. Solution: sample AI routes at 1.0. - -## JavaScript - -```javascript -Sentry.init({ - dsn: process.env.SENTRY_DSN, - tracesSampler: ({ name, attributes, inheritOrSampleWith }) => { - // Standalone gen_ai root spans - if (attributes?.['sentry.op']?.startsWith('gen_ai.') || attributes?.['gen_ai.system']) { - return 1.0; - } - // HTTP routes that trigger AI calls - if (name?.includes('/api/chat') || name?.includes('/api/agent')) { - return 1.0; - } - return inheritOrSampleWith(0.2); // adjust to your baseline - }, -}); -``` - -## Python - -```python -def traces_sampler(sampling_context): - tx = sampling_context.get("transaction_context", {}) - op, name = tx.get("op", ""), tx.get("name", "") - - if op.startswith("gen_ai."): - return 1.0 - if op == "http.server" and any(p in name for p in ["/api/chat", "/api/agent"]): - return 1.0 - - parent = sampling_context.get("parent_sampled") - if parent is not None: - return float(parent) - return 0.2 - -sentry_sdk.init(dsn="...", traces_sampler=traces_sampler) -``` - -If AI is the core product, skip `tracesSampler` and use `tracesSampleRate: 1.0`. - -## Fallback: Metrics + Logs - -If 100% tracing isn't feasible, emit metrics and logs on every LLM call (independent of trace sampling): - -```python -# Metrics - 100% coverage of cost/usage/latency -sentry_sdk.metrics.distribution("gen_ai.token_usage", usage.total_tokens, - attributes={"model": model, "user_id": str(user.id)}) -sentry_sdk.metrics.count("gen_ai.calls", 1, - attributes={"model": model, "status": "error" if error else "success"}) - -# Logs - 100% searchable per-call records -sentry_sdk.logger.info("LLM call", model=model, input_tokens=usage.prompt_tokens, - output_tokens=usage.completion_tokens, latency_ms=response_time_ms) -``` - -JS equivalent uses `Sentry.metrics.*` and `Sentry.logger.*` with the same attribute patterns. - -## Troubleshooting - -| Issue | Solution | -|-------|----------| -| gen_ai spans missing despite sampler returning 1.0 | Parent HTTP transaction was sampled at a lower rate. Add the route to your sampler. | -| `tracesSampler` not called for gen_ai spans | Expected. It only runs on root spans. Sample the parent HTTP route instead. | -| All traces at 100% | Check the fallback rate in `inheritOrSampleWith()` / default return value. |