Skip to content

Commit 3276f64

Browse files
gustavocidornelaswhoseoyster
gustavocidornelas
authored and
whoseoyster
committed
Completes OPEN-5856 Add RAG tracing example
1 parent 6054119 commit 3276f64

File tree

2 files changed

+206
-0
lines changed

2 files changed

+206
-0
lines changed

‎examples/monitoring/llms/rag-tracing/context.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
'Apple was founded as Apple Computer Company on April 1, 1976, by Steve Wozniak, Steve Jobs (1955–2011) and Ronald Wayne to develop and sell Wozniak\'s Apple I personal computer. It was incorporated by Jobs and Wozniak as Apple Computer, Inc. in 1977. The company\'s second computer, the Apple II, became a best seller and one of the first mass-produced microcomputers. Apple went public in 1980 to instant financial success. The company developed computers featuring innovative graphical user interfaces, including the 1984 original Macintosh, announced that year in a critically acclaimed advertisement called "1984". By 1985, the high cost of its products, and power struggles between executives, caused problems. Wozniak stepped back from Apple and pursued other ventures, while Jobs resigned and founded NeXT, taking some Apple employees with him.', 'History\nMain article: History of Apple Inc.\n1976–1980: Founding and incorporation\nSee also: History of Apple Inc. §\xa01971–1985: Jobs and Wozniak\nIn 1976, Steve Jobs co-founded Apple in his parents\' home on Crist Drive in Los Altos, California.[9] Although it is widely believed that the company was founded in the house\'s garage, Apple co-founder Steve Wozniak called it "a bit of a myth".[10] Jobs and Wozniak did, however, move some operations to the garage when the bedroom became too crowded.[11]\nApple\'s first product, the Apple I, designed by Steve Wozniak, was sold as an assembled circuit board and lacked basic features such as a keyboard, monitor, and case. The owner of this unit added a keyboard and wooden case.\nThe Apple II Plus, introduced in 1979, designed primarily by Wozniak', 'Guerrino De Luca\nPaul Deneve\nAl Eisenstat\nTony Fadell\nScott Forstall\nEllen Hancock\nNancy R. Heinen\nRon Johnson\nDavid Nagel\nPeter Oppenheimer\nMark Papermaster\nJon Rubinstein\nBertrand Serlet\nBruce Sewell\nSina Tamaddon\nAvie Tevanian\nSteve Wozniak\nBoard ofdirectorsCurrent\nArthur D. Levinson (Chairman)\nTim Cook (CEO)\nJames A. Bell\nAlex Gorsky\nAl Gore\nAndrea Jung\nRonald D. Sugar\nSusan L. Wagner\nFormer\nMike Markkula (Chairman)\nJohn Sculley (Chairman)\nSteve Jobs (Chairman)\nGil Amelio\nFred D. Anderson\nBill Campbell\nMickey Drexler\nAl Eisenstat\nLarry Ellison\nRobert A. Iger\nDelano Lewis\nArthur Rock\nEric Schmidt\nMichael Scott\nMichael Spindler\nEdgar S. Woolard Jr.\nJerry York\nFounders\nSteve Jobs\nSteve Wozniak\nRonald Wayne', '^ Moses, Asher (October 7, 2011). "Who was Steve Jobs the man?". The Age. Melbourne. Retrieved October 7, 2011.; "Tearful memories for Apple co-founder". The Age. Melbourne. Archived from the original on October 8, 2011. Retrieved October 7, 2011.\n\n^ Flynn, Laurie J. (February 6, 2007). "After Long Dispute, Two Apples Work It Out". The New York Times. Archived from the original on February 7, 2007. Retrieved October 21, 2016.\n\n^ "Wired News: Apple Doin\' the Logo-Motion". September 26, 2003.; "¥ves ฿ennaïm 🌿 (@ZLOK) on Twitter". twitter.com.\n\n^ "Apple Computer". August 27, 1999. Archived from the original on August 27, 1999. Retrieved January 1, 2014.\n\n^ "The Lost Apple Logos You\'ve Never Seen
2+
3+
"Apple also experimented with a number of other unsuccessful consumer targeted products during the 1990s, including digital cameras, portable CD audio players, speakers, video game consoles, the eWorld online service, and TV appliances. Most notably, enormous resources were invested in the problem-plagued Newton tablet division, based on John Sculley's unrealistic market forecasts.[65]\nThroughout this period, Microsoft continued to gain market share with Windows by focusing on delivering software to inexpensive personal computers, while Apple was delivering a richly engineered but expensive experience.[66] Apple relied on high profit margins and never developed a clear response; instead, they sued Microsoft for using a GUI similar to the Apple Lisa in Apple Computer, Inc. v. Microsoft Corp.[67] The lawsuit dragged on for years before it was finally dismissed.", '2007–2011: Success with mobile devices\nNewly announced iPhone on display at the 2007 MacWorld Expo\nDuring his keynote speech at the Macworld Expo on January 9, 2007, Jobs announced that Apple Computer, Inc. would thereafter be known as "Apple Inc.", because the company had shifted its emphasis from computers to consumer electronics.[110] This event also saw the announcement of the iPhone[111] and the Apple TV.[112] The company sold 270,000 iPhone units during the first 30 hours of sales,[113] and the device was called "a game changer for the industry".[114]', 'The success of the lower-cost Macs and PowerBook brought increasing revenue.[59] For some time, Apple was doing incredibly well, introducing fresh new products and generating increasing profits in the process. The magazine MacAddict named the period between 1989 and 1991 as the "first golden age" of the Macintosh.[60]The PenLite is Apple\'s first prototype of a tablet computer. Created in 1992, the project was designed to bring the Mac OS to a tablet – but was canceled in favor of the Newton.[61]\nThe success of Apple\'s lower-cost consumer models, especially the LC, also led to the cannibalization of their higher-priced machines. To address this, management introduced several new brands, selling largely identical machines at different price points, aimed at different markets: the high-end Quadra models, the mid-range Centris line, and the consumer-marketed Performa series. This led to significant market confusion, as customers did not understand the difference between models.[62]', "In May 2001, the company opened its first two Apple Store retail locations in Virginia and California,[91][92] offering an improved presentation of the company's products.[93] At the time, many speculated that the stores would fail,[94] but they went on to become highly successful, and the first of more than 500 stores around the world.[95]\nOn October 23, 2001, Apple debuted the iPod portable digital audio player. The product, which was first sold on November 10, 2001, was phenomenally successful with over 100\xa0million units sold within six years.[96]\nIn 2003, Apple's iTunes Store was introduced. The service offered music downloads for 99¢ a song and integration with the iPod. The iTunes Store quickly became the market leader in online music services, with over five billion downloads by June 19, 2008.[97] Two years later, the iTunes Store was the world's largest music retailer.[98]"
4+
5+
'On December 12, 1980, Apple (ticker symbol "AAPL") went public selling 4.6\xa0million shares at $22 per share ($.10 per share when adjusting for stock splits as of September\xa03, 2022[update]),[21] generating over $100\xa0million, which was more capital than any IPO since Ford Motor Company in 1956.[30] By the end of the day, 300\xa0millionaires were created, from a stock price of $29 per share[31] and a market cap of $1.778\xa0billion.[30][31]', 'As of August\xa03, 2018[update], Apple was the largest publicly traded corporation in the world by market capitalization. On August 2, 2018, Apple became the first publicly traded U.S. company to reach a $1\xa0trillion market value.[284][285] Apple was ranked No. 4 on the 2018 Fortune 500 rankings of the largest United States corporations by total revenue.[286]\nIn July 2022, Apple reported an 11% decline in Q3 profits compared to 2021. Its revenue in the same period rose 2% year-on-year to $83 billion, though this figure was also lower than in 2021, where the increase was at 36%. The general downturn is reportedly caused by the slowing global economy and supply chain disruptions in China.[287]', "Apple became the first publicly traded U.S. company to be valued at over $1\xa0trillion in August 2018, then at $2\xa0trillion in August 2020, and at $3\xa0trillion in January 2022. In June 2023, it was valued at just over $3\xa0trillion.[8] The company receives criticism regarding the labor practices of its contractors, its environmental practices, and its business ethics, including anti-competitive practices and materials sourcing. Nevertheless, the company has a large following and enjoys a high level of brand loyalty. It has also been consistently ranked as one of the world's most valuable brands.", '^ Bagnall, Brian (2005). On the Edge: The Spectacular Rise and Fall of Commodore. Variant Press. pp.\xa0109–112. ISBN\xa0978-0-9738649-0-8.; Personal Computer Market Share: 1975–2004 Archived June 6, 2012, at the Wayback Machine The figures show Mac higher, but that is not a single model.\n\n^ Reimer, Jeremy (December 15, 2005). "Total share: 30 years of personal computer market share figures". Ars Technica. Retrieved September 1, 2023.\n\n^ a b Deffree, Suzanne (December 12, 2018). "Apple IPO makes instant millionaires, December 12, 1980". Retrieved May 16, 2019.\n\n^ a b Dilger, Daniel Eran (December 12, 2013). "Apple, Inc. stock IPO created 300 millionaires 33 years ago today". AppleInsider. Retrieved April 18, 2017.\n\n^ Montag, Ali (May 21, 2018). "Here\'s why your computer has a mouse, according to Steve Jobs in 1985". CNBC. Archived from the original on February 1, 2021. Retrieved May 10, 2021.'

‎examples/monitoring/llms/rag-tracing/rag_tracer.ipynb

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "83c16ef6-98e7-48d0-b82f-4029a730ff00",
6+
"metadata": {},
7+
"source": [
8+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openlayer-ai/examples-gallery/blob/main/monitoring/llms/rag-tracing/rag_tracer.ipynb)\n",
9+
"\n",
10+
"\n",
11+
"# <a id=\"top\">Tracing a RAG system</a>"
12+
]
13+
},
14+
{
15+
"cell_type": "code",
16+
"execution_count": null,
17+
"id": "21137554-ad8e-444b-bf2e-49393f072956",
18+
"metadata": {},
19+
"outputs": [],
20+
"source": [
21+
"import os\n",
22+
"import openai\n",
23+
"\n",
24+
"# OpenAI env variable\n",
25+
"os.environ[\"OPENAI_API_KEY\"] = \"YOUR_OPENAI_KEY_HERE\"\n",
26+
"\n",
27+
"# Openlayer env variables\n",
28+
"os.environ[\"OPENLAYER_API_KEY\"] = \"YOUR_OPENLAYER_API_KEY_HERE\"\n",
29+
"os.environ[\"OPENLAYER_PROJECT_NAME\"] = \"YOUR_OPENLAYER_PROJECT_NAME_HERE\" # Where the traces will be uploaded to"
30+
]
31+
},
32+
{
33+
"cell_type": "markdown",
34+
"id": "20b25a1f-529e-45c5-90e5-26485914f511",
35+
"metadata": {},
36+
"source": [
37+
"## Defining and decorating our RAG system"
38+
]
39+
},
40+
{
41+
"cell_type": "code",
42+
"execution_count": null,
43+
"id": "9e2f8d80-d49a-48f0-8c12-350045dff985",
44+
"metadata": {},
45+
"outputs": [],
46+
"source": [
47+
"%%bash\n",
48+
"\n",
49+
"if [ ! -e \"context.txt\" ]; then\n",
50+
" curl \"https://raw.githubusercontent.com/openlayer-ai/examples-gallery/main/monitoring/llms/rag-tracing/context.txt\" --output \"context.txt\"\n",
51+
"fi"
52+
]
53+
},
54+
{
55+
"cell_type": "code",
56+
"execution_count": null,
57+
"id": "60d470d7-3aa0-4703-a9e7-cab24325a4a5",
58+
"metadata": {},
59+
"outputs": [],
60+
"source": [
61+
"import random\n",
62+
"import time\n",
63+
"\n",
64+
"import numpy as np\n",
65+
"from openai import OpenAI\n",
66+
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
67+
"from sklearn.metrics.pairwise import cosine_similarity\n",
68+
"\n",
69+
"from openlayer import llm_monitors\n",
70+
"from openlayer.tracing import tracer"
71+
]
72+
},
73+
{
74+
"cell_type": "code",
75+
"execution_count": null,
76+
"id": "c8070d3f-ebec-4faf-8959-23e6ac22737d",
77+
"metadata": {},
78+
"outputs": [],
79+
"source": [
80+
"class RagPipeline:\n",
81+
" def __init__(self, context_path: str):\n",
82+
" # Wrap OpenAI client with Openlayer's OpenAIMonitor to trace it \n",
83+
" self.openai_client = OpenAI()\n",
84+
" llm_monitors.OpenAIMonitor(client=self.openai_client)\n",
85+
" \n",
86+
" self.vectorizer = TfidfVectorizer()\n",
87+
" with open(context_path, 'r', encoding='utf-8') as file:\n",
88+
" self.context_sections = file.read().split('\\n\\n') \n",
89+
" self.tfidf_matrix = self.vectorizer.fit_transform(self.context_sections)\n",
90+
"\n",
91+
" # Decorate the functions you'd like to trace with @tracer.trace()\n",
92+
" @tracer.trace()\n",
93+
" def query(self, user_query: str) -> str:\n",
94+
" \"\"\"Main method.\n",
95+
"\n",
96+
" Answers to a user query with the LLM.\n",
97+
" \"\"\"\n",
98+
" context = self.retrieve_context(user_query)\n",
99+
" prompt = self.inject_prompt(user_query, context)\n",
100+
" answer = self.generate_answer_with_gpt(prompt)\n",
101+
" return answer\n",
102+
"\n",
103+
" @tracer.trace()\n",
104+
" def retrieve_context(self, query: str) -> str:\n",
105+
" \"\"\"Context retriever. \n",
106+
" \n",
107+
" Given the query, returns the most similar context (using TFIDF).\n",
108+
" \"\"\"\n",
109+
" query_vector = self.vectorizer.transform([query])\n",
110+
" cosine_similarities = cosine_similarity(query_vector, self.tfidf_matrix).flatten()\n",
111+
" most_relevant_idx = np.argmax(cosine_similarities)\n",
112+
" return self.context_sections[most_relevant_idx]\n",
113+
"\n",
114+
" @tracer.trace()\n",
115+
" def inject_prompt(self, query: str, context: str):\n",
116+
" \"\"\"Combines the query with the context and returns\n",
117+
" the prompt (formatted to conform with OpenAI models).\"\"\"\n",
118+
" return [\n",
119+
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
120+
" {\"role\": \"user\", \"content\": f\"Answer the user query using only the following context: {context}. \\nUser query: {query}\"}\n",
121+
" ]\n",
122+
"\n",
123+
" @tracer.trace()\n",
124+
" def generate_answer_with_gpt(self, prompt):\n",
125+
" \"\"\"Forwards the prompt to GPT and returns the answer.\"\"\"\n",
126+
" response = self.openai_client.chat.completions.create(\n",
127+
" messages=prompt,\n",
128+
" model=\"gpt-3.5-turbo\",\n",
129+
" )\n",
130+
" return response.choices[0].message.content.strip()"
131+
]
132+
},
133+
{
134+
"cell_type": "code",
135+
"execution_count": null,
136+
"id": "f96f7073-7be4-4254-a6c9-eb808312beb8",
137+
"metadata": {},
138+
"outputs": [],
139+
"source": [
140+
"rag = RagPipeline(\"context.txt\")"
141+
]
142+
},
143+
{
144+
"cell_type": "code",
145+
"execution_count": null,
146+
"id": "50e046fd-68f1-4f66-b2a1-03aa95b9b367",
147+
"metadata": {},
148+
"outputs": [],
149+
"source": [
150+
"rag.query(\"Who were the founders of Apple?\")"
151+
]
152+
},
153+
{
154+
"cell_type": "code",
155+
"execution_count": null,
156+
"id": "afc7f963-fc13-4e93-b3ef-98aa183770a3",
157+
"metadata": {},
158+
"outputs": [],
159+
"source": [
160+
"rag.query(\"When did Apple IPO?\")"
161+
]
162+
},
163+
{
164+
"cell_type": "markdown",
165+
"id": "42f1e832-4c3f-4a6a-8013-8607ff141f67",
166+
"metadata": {},
167+
"source": [
168+
"That's it! After each inference, the traces are uploaded to Openlayer. If you navigate to your project, you should see the traces for these two inferences with our RAG system."
169+
]
170+
},
171+
{
172+
"cell_type": "code",
173+
"execution_count": null,
174+
"id": "f960a36f-3438-4c81-8cdb-ca078aa509cd",
175+
"metadata": {},
176+
"outputs": [],
177+
"source": []
178+
}
179+
],
180+
"metadata": {
181+
"kernelspec": {
182+
"display_name": "Python 3 (ipykernel)",
183+
"language": "python",
184+
"name": "python3"
185+
},
186+
"language_info": {
187+
"codemirror_mode": {
188+
"name": "ipython",
189+
"version": 3
190+
},
191+
"file_extension": ".py",
192+
"mimetype": "text/x-python",
193+
"name": "python",
194+
"nbconvert_exporter": "python",
195+
"pygments_lexer": "ipython3",
196+
"version": "3.9.18"
197+
}
198+
},
199+
"nbformat": 4,
200+
"nbformat_minor": 5
201+
}

0 commit comments

Comments
 (0)