|
| 1 | +# Monocle User Guide |
| 2 | + |
| 3 | +## Monocle Concepts |
| 4 | +### Traces |
| 5 | +Traces are the full view of a single end-to-end application KPI, for example Chatbot application to provide a response to end user’s question. Traces consist of various metadata about the application run including status, start time, duration, input/outputs etc. They also include a list of individual steps aka “spans with details about that step. |
| 6 | +It’s typically the workflow code components of an application that generate the traces for application runs. |
| 7 | +### Spans |
| 8 | +Spans are the individual steps executed by the application to perform a GenAI related task”, for example app retrieving vectors from DB, app querying LLM for inference etc. The span includes the type of operation, start time, duration and metadata relevant to that step e.g., Model name, parameters and model endpoint/server for an inference request. |
| 9 | +It’s typically the workflow code components of an application that generate the traces for application runs. |
| 10 | + |
| 11 | +## Setup Monocle |
| 12 | +- You can download Monocle library releases from Pypi |
| 13 | +``` |
| 14 | + > pip install monocle_apptrace |
| 15 | +``` |
| 16 | + |
| 17 | +- For Azure support (to upload traces to Azure), install with the azure extra: |
| 18 | +``` |
| 19 | + > pip install monocle_apptrace[azure] |
| 20 | +``` |
| 21 | + |
| 22 | +- For AWS support (to upload traces to AWS), install with the aws extra: |
| 23 | +``` |
| 24 | + > pip install monocle_apptrace[aws] |
| 25 | +``` |
| 26 | + |
| 27 | +- You can locally build and install Monocle library from source |
| 28 | +``` |
| 29 | + > pip install . |
| 30 | +``` |
| 31 | +- Install the optional test dependencies listed against dev in pyproject.toml in editable mode |
| 32 | +``` |
| 33 | + > pip install -e ".[dev]" |
| 34 | +``` |
| 35 | + |
| 36 | + |
| 37 | +## Using Monocle with your application to generate traces |
| 38 | +### Enable Monocle tracing |
| 39 | +You need to import monocle package and invoke the API ``setup_monocle_telemetry(workflow=<workflow-name>)`` to enable the tracing. The 'workflow-name' is what you define to identify the give application workflow, for example "customer-chatbot". Monocle trace will include this name in every trace. The trace output will include a list of spans in the traces. You can print the output on the console or send it to an HTTP endpoint. |
| 40 | + |
| 41 | +### Using Monocle's out of box support of genAI technology components |
| 42 | +Monocle community has done the hard work of figuring out what to trace and how to extract relevant details from multiple genAI technology components. For example, if you have a python app coded using LlamaIndex and using models hostsed in OpenAI, Monocle can seamlessly trace your app. All you need to do enable Monocle tracing. |
| 43 | + |
| 44 | +### Using Monocle's Support for Adding Custom Attributes |
| 45 | +Monocle provides users with the ability to add custom attributes to various spans, such as inference and retrieval spans, by utilizing the output processor within its metamodel. This feature allows for dynamic attribute assignment through lambda functions, which operate on an arguments dictionary. |
| 46 | +The arguments dictionary contains key-value pairs that can be used to compute custom attributes. The dictionary includes the following components: |
| 47 | +```python |
| 48 | +arguments = {"instance":instance, "args":args, "kwargs":kwargs, "output":return_value} |
| 49 | +``` |
| 50 | +By leveraging this dictionary, users can define custom attributes for spans, enabling the integration of additional context and information into the tracing process. The lambda functions used in the attributes field can access and process these values to enrich the span with relevant custom data. |
| 51 | + |
| 52 | +#### Example - Enable Monocle tracing in your application |
| 53 | +```python |
| 54 | +from monocle_apptrace.instrumentor import setup_monocle_telemetry |
| 55 | +from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter |
| 56 | +from langchain.chains import LLMChain |
| 57 | +from langchain_openai import OpenAI |
| 58 | +from langchain.prompts import PromptTemplate |
| 59 | + |
| 60 | +# Call the setup Monocle telemetry method |
| 61 | +setup_monocle_telemetry(workflow_name = "simple_math_app") |
| 62 | + |
| 63 | +llm = OpenAI() |
| 64 | +prompt = PromptTemplate.from_template("1 + {number} = ") |
| 65 | + |
| 66 | +chain = LLMChain(llm=llm, prompt=prompt) |
| 67 | +chain.invoke({"number":2}) |
| 68 | + |
| 69 | +# Request callbacks: Finally, let's use the request `callbacks` to achieve the same result |
| 70 | +chain = LLMChain(llm=llm, prompt=prompt) |
| 71 | +chain.invoke({"number":2}, {"callbacks":[handler]}) |
| 72 | + |
| 73 | +``` |
| 74 | + |
| 75 | +### Accessing monocle trace |
| 76 | +By default monocle generate traces in a json file created in the local directory where the application is running. The file name by default is monocle_trace_{workflow_name}\_{trace_id}\_{timestamp}.json where the trace_id is a unique number generated by monocle for every trace. Please refere to [Trace span json](Monocle_User_Guide.md#trace-span-json). The file path and format can be changed by setting those properties as argement to ```setup_monocle_telemetry()```. For example, |
| 77 | +``` |
| 78 | +setup_monocle_telemetry(workflow_name = "simple_math_app", |
| 79 | + span_processors=[BatchSpanProcessor(FileSpanExporter( |
| 80 | + out_path = "/tmp", |
| 81 | + file_prefix = "map_app_prod_trace_", |
| 82 | + time_format = "%Y-%m-%d")) |
| 83 | + ]) |
| 84 | +``` |
| 85 | +To print the trace on the console, use ```ConsoleSpanExporter()``` instead of ```FileSpanExporter()``` |
| 86 | + |
| 87 | +For Azure: |
| 88 | + Install the Azure support as shown in the setup section, then use ```AzureBlobSpanExporter()``` to upload the traces to Azure. |
| 89 | + |
| 90 | +For AWS: |
| 91 | + Install the AWS support as shown in the setup section, then use ```S3SpanExporter()``` to upload the traces to an S3 bucket. |
| 92 | + |
| 93 | +### Leveraging Monocle's extensibility to handle customization |
| 94 | +When the out of box features from app frameworks are not sufficent, the app developers have to add custom code. For example, if you are extending a LLM class in LlamaIndex to use a model hosted in NVIDIA Triton. This new class is not know to Monocle. You can specify this new class method part of Monocle enabling API and it will be able to trace it. |
| 95 | + |
| 96 | +#### Default configuration of instrumented methods in Monocle |
| 97 | +The following files comprise of default configuration of instrumented methods and span names corresponding to them, for each framework respectively. |
| 98 | +- [src/monocle_apptrace/langchain/__init__.py](src/monocle_apptrace/langchain/__init__.py), |
| 99 | +- [src/monocle_apptrace/llamaindex/__init__.py](src/monocle_apptrace/llamaindex/__init__.py), |
| 100 | +- [src/monocle_apptrace/haystack/__init__.py](src/monocle_apptrace/haystack/__init__.py) |
| 101 | + |
| 102 | +Following configuration instruments ```invoke(..)``` of ```RunnableSequence```, aka chain or worflow in Langchain parlance, to emit the span. |
| 103 | +``` |
| 104 | + { |
| 105 | + "package": "langchain.schema.runnable", |
| 106 | + "object": "RunnableSequence", |
| 107 | + "method": "invoke", |
| 108 | + "span_name": "langchain.workflow", |
| 109 | + "wrapper": task_wrapper |
| 110 | + } |
| 111 | +``` |
| 112 | +#### Example - Monitoring custom methods with Monocle |
| 113 | +```python |
| 114 | +from monocle_apptrace.wrapper import WrapperMethod,task_wrapper,atask_wrapper |
| 115 | +from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter |
| 116 | + |
| 117 | +# extend the default wrapped methods list as follows |
| 118 | +app_name = "simple_math_app" |
| 119 | +setup_monocle_telemetry( |
| 120 | + workflow_name=app_name, |
| 121 | + span_processors=[BatchSpanProcessor(ConsoleSpanExporter())], |
| 122 | + wrapper_methods=[ |
| 123 | + WrapperMethod( |
| 124 | + package="langchain.schema.runnable", |
| 125 | + object_name="RunnableParallel", |
| 126 | + method="invoke", |
| 127 | + span_name="langchain.workflow", |
| 128 | + wrapper=task_wrapper), |
| 129 | + WrapperMethod( |
| 130 | + package="langchain.schema.runnable", |
| 131 | + object_name="RunnableParallel", |
| 132 | + method="ainvoke", |
| 133 | + span_name="langchain.workflow", |
| 134 | + wrapper=atask_wrapper) |
| 135 | + ]) |
| 136 | + |
| 137 | +``` |
| 138 | + |
| 139 | +### Going beyond supported genAI components |
| 140 | +- If you are using an application framework, model hosting service/infra etc. that's not currently supported by Monocle, please submit a github issue to add that support. |
| 141 | +- Monocle community is working on adding an SDK to enable applications to generate their own traces. |
| 142 | + |
| 143 | +## Understanding the trace output |
| 144 | + |
| 145 | +### Trace span json |
| 146 | + |
| 147 | +Monocle generates spans which adhere to [Tracing API | OpenTelemetry](https://opentelemetry.io/docs/specs/otel/trace/api/#span) format. The trace output is an array of spans. Each trace has a unique id. Every span has in the trace has this parent ```trace_id```. Please note that ```trace_id``` groups related spans and is auto generated with-in Monocle. |
| 148 | + |
| 149 | +| Span JSON | Description | |
| 150 | +| ------------- | ------------- | |
| 151 | +| {|| |
| 152 | +| "```name```": "langchain.workflow",|span name and is configurable in [__init.py__](src/monocle_apptrace/langchain/__init__.py) or in ```setup_monocle_telemetry(...)```| |
| 153 | +| "```context```": {|this gets autogenerated| |
| 154 | +|   "```trace_id```": "0xe5269f0e534efa098b240f974220d6b7",|| |
| 155 | +|   "```span_id```": "0x30b13075eca52f44",|| |
| 156 | +|   "```trace_state```": "[]"|| |
| 157 | +|   },|| |
| 158 | +|"```kind```": "SpanKind.INTERNAL",| an enum that describes what this span is about. Default value is SpanKind.INTERNAL, as current enums do not cover ML apps | |
| 159 | +|"```parent_id```": null,|if null, this is root span| |
| 160 | +|"```start_time```": "2024-07-16T17:05:15.544861Z",|| |
| 161 | +|"```end_time```": "2024-07-16T17:05:43.502007Z",|| |
| 162 | +|"```status```": {|| |
| 163 | +|  "```status_code```": "UNSET"| status of span to OK or ERROR. Default is UNSET| |
| 164 | +|  },|| |
| 165 | +|"```attributes```": {|| |
| 166 | +|  "workflow_name": "ml_rag_app",|defines the name of the service being set in ```setup_monocle_telemetry(...)``` during initialization of instrumentation| |
| 167 | +|  "workflow_type": "workflow.langchain"|type of framework that generated this span| |
| 168 | +|  },|| |
| 169 | +|"```events```": [|captures the log records| |
| 170 | +|  {|| |
| 171 | +|   "```name```": "input",|name of the event. If the span is about LLM, then this will be 'input'. For vector store retrieval, this would be 'context_input'| |
| 172 | +|   "```timestamp```": "2024-07-16T17:05:15.544874Z",|| |
| 173 | +|   "```attributes```": {|captures the 'input' attributes. Based on the workflow of the ML framework being used, the attributes change| |
| 174 | +|    "question": "What is Task Decomposition?",|represents LLM query| |
| 175 | +|    "q_a_pairs": "..." |represents questions and answers for a few shot LLM prompting | |
| 176 | +|   }|| |
| 177 | +|  },|| |
| 178 | +|  {|| |
| 179 | +|   "```name```": "output",|represents 'ouput' event of LLM| |
| 180 | +|   "```timestamp```": "2024-07-16T17:05:43.501996Z",|| |
| 181 | +|  "```attributes```": {|| |
| 182 | +|    "response": "Task Decomposition is ..."|response to LLM query. | |
| 183 | +|   }|| |
| 184 | +|  }|| |
| 185 | +|  ],|| |
| 186 | +|  "```links```": [],|unused. Ideally this links other causally-related spans,<br/> but as spans are grouped by ```trace_id```, and ```parent_id``` links to parent span, this is unused| |
| 187 | +|  "```resource```": {|represents the service name or server or machine or container which generated the span| |
| 188 | +|    "```attributes```": {|| |
| 189 | +|     "service.name": "ml_rag_app"|only service.name is being populated and defaults to the value of 'workflow_name' | |
| 190 | +|    },|| |
| 191 | +|  "```schema_url```": ""|unused| |
| 192 | +|   }|| |
| 193 | +|} | | |
0 commit comments