-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Add Datadog Integration Documentation #3642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sabrenner
wants to merge
11
commits into
crewAIInc:main
Choose a base branch
from
sabrenner:sabrenner/datadog-integration
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+318
−0
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
1e65afa
add datadog llm observability integration guide
sabrenner 57cc464
spacing fix
sabrenner c54358f
wording changes
sabrenner 40eb5b0
alphabetize docs listing
sabrenner 2b5de45
Merge branch 'main' of github.com:sabrenner/crewAI into sabrenner/dat…
sabrenner c1bd9db
Update docs/en/observability/datadog.mdx
sabrenner ad04b1e
Merge branch 'main' into sabrenner/datadog-integration
greysonlalonde f23e1e6
Merge branch 'main' into sabrenner/datadog-integration
greysonlalonde d0af6b8
add translations
sabrenner 5e2421f
Merge branch 'sabrenner/datadog-integration' of github.com:sabrenner/…
sabrenner 0a73f19
fix korean code block
sabrenner File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
--- | ||
title: Datadog Integration | ||
description: Learn how to integrate Datadog with CrewAI to submit LLM Observability traces to Datadog. | ||
icon: dog | ||
mode: "wide" | ||
--- | ||
|
||
# Integrate Datadog with CrewAI | ||
|
||
This guide will demonstrate how to integrate **[Datadog LLM Observability](https://docs.datadoghq.com/llm_observability/)** with **CrewAI** using [Datadog auto-instrumentation](https://docs.datadoghq.com/llm_observability/instrumentation/auto_instrumentation?tab=python). By the end of this guide, you will be able to submit LLM Observability traces to Datadog and view your CrewAI agent runs in Datadog LLM Observability's [Agentic Execution View](https://docs.datadoghq.com/llm_observability/monitoring/agent_monitoring). | ||
|
||
## What is Datadog LLM Observability? | ||
|
||
[Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) helps AI engineers, data scientists, and application developers quickly develop, evaluate, and monitor LLM applications. Confidently improve output quality, performance, costs, and overall risk with structured experiments, end-to-end tracing across AI agents, and evaluations. | ||
|
||
## Getting Started | ||
|
||
### Install Dependencies | ||
|
||
```shell | ||
pip install ddtrace crewai crewai-tools | ||
``` | ||
|
||
### Set Environment Variables | ||
|
||
If you do not have a Datadog API key, you can [create an account](https://www.datadoghq.com/) and [get your API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys). | ||
|
||
You will also need to specify an ML Application name in the following environment variables. An ML Application is a grouping of LLM Observability traces associated with a specific LLM-based application. See [ML Application Naming Guidelines](https://docs.datadoghq.com/llm_observability/instrumentation/sdk?tab=python#application-naming-guidelines) for more information on limitations with ML Application names. | ||
|
||
```shell | ||
export DD_API_KEY=<YOUR_DD_API_KEY> | ||
export DD_SITE=<YOUR_DD_SITE> | ||
export DD_LLMOBS_ENABLED=true | ||
export DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> | ||
export DD_LLMOBS_AGENTLESS_ENABLED=true | ||
export DD_APM_TRACING_ENABLED=false | ||
``` | ||
|
||
Additionally, configure any LLM provider API keys | ||
|
||
```shell | ||
export OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> | ||
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY> | ||
export GEMINI_API_KEY=<YOUR_GEMINI_API_KEY> | ||
... | ||
``` | ||
|
||
### Create a CrewAI Agent Application | ||
|
||
```python | ||
# crewai_agent.py | ||
from crewai import Agent, Task, Crew | ||
|
||
from crewai_tools import ( | ||
WebsiteSearchTool | ||
) | ||
|
||
web_rag_tool = WebsiteSearchTool() | ||
|
||
writer = Agent( | ||
role="Writer", | ||
goal="You make math engaging and understandable for young children through poetry", | ||
backstory="You're an expert in writing haikus but you know nothing of math.", | ||
tools=[web_rag_tool], | ||
) | ||
|
||
task = Task( | ||
description=("What is {multiplication}?"), | ||
expected_output=("Compose a haiku that includes the answer."), | ||
agent=writer | ||
) | ||
|
||
crew = Crew( | ||
agents=[writer], | ||
tasks=[task], | ||
share_crew=False | ||
) | ||
|
||
output = crew.kickoff(dict(multiplication="2 * 2")) | ||
``` | ||
|
||
### Run the Application with Datadog Auto-Instrumentation | ||
|
||
With the [environment variables](#set-environment-variables) set, you can now run the application with Datadog auto-instrumentation. | ||
|
||
```shell | ||
ddtrace-run python crewai_agent.py | ||
``` | ||
|
||
### View the Traces in Datadog | ||
|
||
After running the application, you can view the traces in [Datadog LLM Observability's Traces View](https://app.datadoghq.com/llm/traces), selecting the ML Application name you chose from the top-left dropdown. | ||
|
||
Clicking on a trace will show you the details of the trace, including total tokens used, number of LLM calls, models used, and estimated cost. Clicking into a specific span will narrow down these details, and show related input, output, and metadata. | ||
|
||
 | ||
|
||
Additionally, you can view the execution graph view of the trace, which shows the control and data flow of the trace, which will scale with larger agents to show handoffs and relationships between LLM calls, tool calls, and agent interactions. | ||
|
||
 | ||
|
||
## References | ||
|
||
- [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) | ||
- [Datadog LLM Observability CrewAI Auto-Instrumentation](https://docs.datadoghq.com/llm_observability/instrumentation/auto_instrumentation?tab=python#crew-ai) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
--- | ||
title: Datadog 통합 | ||
description: Datadog을 CrewAI와 통합하여 LLM Observability 트레이스들을 Datadog에 제출하는 방법을 알아보세요. | ||
icon: dog | ||
mode: "wide" | ||
--- | ||
|
||
# Datadog을 CrewAI와 통합하기 | ||
|
||
이 가이드에서는 Datadog 자동 계측을 사용하여 **Datadog**을 **CrewAI**와 통합하는 방법을 보여드립니다. 이 가이드가 끝나면 LLM Observability 트레이스를 Datadog에 제출하고 CrewAI 에이전트 실행을 Datadog LLM Observability의 에이전트 실행 보기에서 볼 수 있게 됩니다. | ||
|
||
## Datadog LLM Observability란 무엇인가요? | ||
|
||
[Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/)는 AI 엔지니어, 데이터 과학자, 애플리케이션 개발자가 LLM 애플리케이션을 신속하게 개발, 평가, 모니터링할 수 있도록 도와줍니다. 구조화된 실험, AI 에이전트 전반의 엔드투엔드 추적, 평가를 통해 결과물 품질, 성능, 비용, 전반적인 위험을 확실하게 개선할 수 있습니다. | ||
|
||
## 시작하기 | ||
|
||
### 설치 종속성 | ||
|
||
```shell | ||
pip install ddtrace crewai crewai-tools | ||
``` | ||
|
||
### 환경 변수 설정하기 | ||
|
||
Datadog API 키가 없는 경우, [계정 만들기](https://www.datadoghq.com/) 및 [API 키 받기](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys)를 할 수 있습니다. | ||
|
||
또한 다음 환경 변수에 ML 애플리케이션 이름을 지정해야 합니다. ML 애플리케이션은 특정 LLM 기반 애플리케이션과 관련된 LLM Observability 트레이스의 그룹입니다. ML 애플리케이션 이름 제한에 대한 자세한 내용은 [ML 애플리케이션 이름 지정 가이드라인](https://docs.datadoghq.com/llm_observability/instrumentation/sdk?tab=python#application-naming-guidelines)을 참조하세요. | ||
|
||
```shell | ||
export DD_API_KEY=<YOUR_DD_API_KEY> | ||
export DD_SITE=<YOUR_DD_SITE> | ||
export DD_LLMOBS_ENABLED=true | ||
export DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> | ||
export DD_LLMOBS_AGENTLESS_ENABLED=true | ||
export DD_APM_TRACING_ENABLED=false | ||
``` | ||
|
||
또한 LLM 공급자 API 키를 설정합니다. | ||
|
||
```shell | ||
export OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> | ||
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY> | ||
export GEMINI_API_KEY=<YOUR_GEMINI_API_KEY> | ||
... | ||
``` | ||
|
||
### 크루AI 에이전트 애플리케이션 생성하기 | ||
|
||
```python | ||
# crewai_agent.py | ||
from crewai import Agent, Task, Crew | ||
|
||
from crewai_tools import ( | ||
WebsiteSearchTool | ||
) | ||
|
||
web_rag_tool = WebsiteSearchTool() | ||
|
||
writer = Agent( | ||
role="작가", | ||
goal="시를 통해 어린이들이 수학을 흥미롭고 이해하기 쉽게 설명합니다", | ||
backstory="당신은 하이쿠를 쓰는 전문가이지만 수학은 전혀 모릅니다.", | ||
tools=[web_rag_tool], | ||
) | ||
|
||
task = Task( | ||
description=("{곱셈}이란 무엇인가요?"), | ||
expected_output=("답을 포함하는 하이쿠를 작성하세요."), | ||
agent=writer | ||
) | ||
|
||
crew = Crew( | ||
agents=[writer], | ||
tasks=[task], | ||
share_crew=False | ||
) | ||
|
||
output = crew.kickoff(dict(곱셈="2 * 2")) | ||
``` | ||
|
||
### Datadog 자동 계측을 사용하여 애플리케이션 실행하기 | ||
|
||
[환경 변수](#환경-변수-설정하기)를 설정하면 이제 Datadog 자동 계측을 통해 애플리케이션을 실행할 수 있습니다. | ||
|
||
```shell | ||
ddtrace-run python crewai_agent.py | ||
``` | ||
|
||
### Datadog에서 트레이스 추적하기 | ||
|
||
애플리케이션을 실행한 후 왼쪽 상단 드롭다운에서 선택한 ML 애플리케이션 이름을 선택하면 [Datadog LLM Observability의 트레이스 보기](https://app.datadoghq.com/llm/traces)에서 트레이스들을 확인할 수 있습니다. | ||
|
||
트레이스를 클릭하면 사용된 총 토큰, LLM 호출 수, 사용된 모델, 예상 비용 등 트레이스에 대한 세부 정보가 표시됩니다. 특정 스팬(span)을 클릭하면 이러한 세부 정보의 범위가 좁혀지고 관련 입력, 출력 및 메타데이터가 표시됩니다. | ||
|
||
 | ||
|
||
또한, 트레이스의 제어 및 데이터 흐름을 보여주는 트레이스의 실행 그래프 보기를 볼 수 있으며, 이는 더 큰 에이전트로 확장하여 LLM 호출, 도구 호출 및 에이전트 상호 작용 간의 핸드오프와 관계를 보여줍니다. | ||
|
||
 | ||
|
||
## 참조 | ||
|
||
- [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) | ||
- [Datadog LLM 옵저버빌리티 크루AI 자동 계측](https://docs.datadoghq.com/llm_observability/instrumentation/auto_instrumentation?tab=python#crew-ai) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
--- | ||
title: Integração Datadog | ||
description: Saiba como integrar o Datadog com o CrewAI para enviar os rastros de observabilidade do LLM para o Datadog. | ||
icon: dog | ||
mode: "wide" | ||
--- | ||
|
||
# Integrar o Datadog com a CrewAI | ||
|
||
Este guia demonstrará como integrar o **Datadog** com o **CrewAI** usando a instrumentação automática do Datadog. Ao final deste guia, você poderá enviar rastreamentos do LLM Observability para o Datadog e visualizar as execuções do agente CrewAI no Agentic Execution View do Datadog LLM Observability. | ||
|
||
## O que é o Datadog LLM Observability? | ||
|
||
O [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) ajuda os engenheiros de IA, cientistas de dados e desenvolvedores de aplicativos a desenvolver, avaliar e monitorar rapidamente os aplicativos LLM. Melhore com confiança a qualidade dos resultados, o desempenho, os custos e o risco geral com experimentos estruturados, rastreamento de ponta a ponta em agentes de IA e avaliações. | ||
|
||
## Primeiros passos | ||
|
||
### Instalar dependências | ||
|
||
```shell | ||
pip install ddtrace crewai crewai-tools | ||
``` | ||
|
||
### Definir variáveis de ambiente | ||
|
||
Se você não tiver uma chave de API da Datadog, poderá [criar uma conta](https://www.datadoghq.com/) e [obter sua chave de API](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys). | ||
|
||
Você também precisará especificar um nome de aplicativo de ML nas seguintes variáveis de ambiente. Um aplicativo de ML é um agrupamento de rastros de observabilidade do LLM associados a um aplicativo específico baseado em LLM. Consulte [ML Application Naming Guidelines](https://docs.datadoghq.com/llm_observability/instrumentation/sdk?tab=python#application-naming-guidelines) para obter mais informações sobre as limitações dos nomes de aplicativos do ML. | ||
|
||
```shell | ||
export DD_API_KEY=<SUA_API_KEY> | ||
export DD_SITE=<Seu_DD_SITE> | ||
export DD_LLMOBS_ENABLED=true | ||
export DD_LLMOBS_ML_APP=<YOUR_ML_APP_NAME> | ||
export DD_LLMOBS_AGENTLESS_ENABLED=true | ||
export DD_APM_TRACING_ENABLED=false | ||
``` | ||
|
||
Além disso, configure todas as chaves de API do provedor LLM | ||
|
||
```shell | ||
export OPENAI_API_KEY=<SUA_OPENAI_API_KEY> | ||
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY> | ||
export GEMINI_API_KEY=<YOUR_GEMINI_API_KEY> | ||
... | ||
``` | ||
|
||
### Criar um aplicativo agente CrewAI | ||
|
||
```python | ||
# crewai_agent.py | ||
from crewai import Agent, Task, Crew | ||
|
||
from crewai_tools import ( | ||
WebsiteSearchTool | ||
) | ||
|
||
web_rag_tool = WebsiteSearchTool() | ||
|
||
writer = Agent( | ||
role="Writer", | ||
goal="Você torna a matemática envolvente e compreensível para crianças pequenas por meio da poesia", | ||
backstory="Você é um especialista em escrever haikus, mas não sabe nada de matemática", | ||
tools=[web_rag_tool], | ||
) | ||
|
||
task = Task( | ||
description=("O que é {multiplicação}?"), | ||
expected_output=("Componha um haicai que inclua a resposta."), | ||
agent=writer | ||
) | ||
|
||
crew = Crew( | ||
agents=[writer], | ||
tasks=[task], | ||
share_crew=False | ||
) | ||
|
||
output = crew.kickoff(dict(multiplicação="2 * 2")) | ||
``` | ||
|
||
### Executar o aplicativo com a instrumentação automática do Datadog | ||
|
||
Com as [variáveis de ambiente](#definir-variáveis-de-ambiente) definidas, agora você pode executar o aplicativo com a instrumentação automática do Datadog. | ||
|
||
```shell | ||
ddtrace-run python crewai_agent.py | ||
``` | ||
|
||
### Visualizar os rastros no Datadog | ||
|
||
Depois de executar o aplicativo, você pode visualizar os traços na [Datadog LLM Observability's Traces View](https://app.datadoghq.com/llm/traces), selecionando o nome do aplicativo de ML que você escolheu no menu suspenso superior esquerdo. | ||
|
||
Ao clicar em um rastreamento, você verá os detalhes do rastreamento, incluindo o total de tokens usados, o número de chamadas LLM, os modelos usados e o custo estimado. Clicar em um intervalo específico reduzirá esses detalhes e mostrará a entrada, a saída e os metadados relacionados. | ||
|
||
 | ||
|
||
Além disso, você pode visualizar a visualização do gráfico de execução do rastreamento, que mostra o controle e o fluxo de dados do rastreamento, que será dimensionado com agentes maiores para mostrar transferências e relacionamentos entre chamadas LLM, chamadas de ferramentas e interações de agentes. | ||
|
||
 | ||
|
||
## Referências | ||
|
||
- [Datadog LLM Observability](https://www.datadoghq.com/product/llm-observability/) | ||
- [Datadog LLM Observability CrewAI Auto-Instrumentation](https://docs.datadoghq.com/llm_observability/instrumentation/auto_instrumentation?tab=python#crew-ai) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.