-
Notifications
You must be signed in to change notification settings - Fork 202
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* creating quickstart notebook for appui * fixing some runner bugs and adding info to quickstart * add screenshots to repo * clear output * unneeded try
- Loading branch information
Showing
9 changed files
with
285 additions
and
49 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,255 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Running apps in the dashboard\n", | ||
| "\n", | ||
| "This notebook describes how to run your apps from the streamlit dashboard. Following this notebook, you should be able to access your apps and interact with them within the streamlit dashboard under the **Apps** page (see screenshot below). Make sure to check the **Setting up** section below to get your app in the list of apps on that page.\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "Clicking *New session* under any of these apps will bring up an empty transcript of the interactions between the user (you) and the app (see screenshot below). Typing a message under *Your message* on the bottom of the window, and pressing enter, will run your app with that specified message as input, produce the app output, and add both to the chat transcript under the *Records* column.\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "Several other inputs are present on this page which control what about the produced transcript record to show alongside their inputs/outputs.\n", | ||
| "\n", | ||
| "- Under the *App details* heading, you can specify Selectors of components of your app which then shows them in that column as the transcript is produced. These selectors are the same specifications as seen in the green labels in other parts of the Dashboard. \n", | ||
| "\n", | ||
| "- Under the *Records* heading, you can add Selectors of record parts in a similar manner. Each added selectors will then be presented alongside each input-output pair in the transcript.\n", | ||
| "\n", | ||
| "Note: When specifying selectors, you skip the \"Select.App\" or \"Select.Record\" part of those selectors. Also the \"RecordInput\" and \"RecordOutput\" (not that you would need them given they are in the transcript already) are specified as \"main_input\" and \"main_output\", respectively. \n", | ||
| "\n", | ||
| "An example of a running session with several selectors is shown in the following screenshot:\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "The session is preserved when navigating away from this page, letting you inspect the produced records in the **Evaluation** page, for example. To create a new session, you first need to end the existing one by pressing the \"End session\" button on top of the runner page." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Setting up" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### App loader\n", | ||
| "\n", | ||
| "To be able to create a new session or \"conversation\", we need to be able to\n", | ||
| "reset the langchain app to its initial state. For this purpose, we require the\n", | ||
| "callable that produces a new chain that is configured for the start of the\n", | ||
| "conversation. Things like memory or other stateful aspects of the chain should\n", | ||
| "be at their initial values. Because of this, we need to construct all components\n", | ||
| "that could theoretically be stateful fully inside the required callable.\n", | ||
| "\n", | ||
| "**NOTE**: We impose a limit on how big the serialization of the loader is. To\n", | ||
| "reduce its size, do not rely on globals defined outside of the function to\n", | ||
| "implement its functionality. The llama_index example in this notebook shows a\n", | ||
| "case where it may be a good idea to include a global (i.e. something downloaded\n", | ||
| "from the web). \n", | ||
| "\n", | ||
| "**WARNING**: This function needs to return a new instance of the app independent\n", | ||
| "of any others produced earlier. That is, you cannot take an existing or\n", | ||
| "pre-loaded app, clear its memory, and return it. As part of the dashboard,\n", | ||
| "multiple instances of an app need to operate at the same time without\n", | ||
| "interference in their states." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## langchain example" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "def load_langchain_app():\n", | ||
| " # All relevant imports must be inside this function.\n", | ||
| "\n", | ||
| " from langchain.llms import OpenAI\n", | ||
| " from langchain.chains import ConversationChain\n", | ||
| " from langchain.memory import ConversationSummaryBufferMemory\n", | ||
| "\n", | ||
| " llm = OpenAI(temperature=0.9, max_tokens=128)\n", | ||
| "\n", | ||
| " # Conversation memory.\n", | ||
| " memory = ConversationSummaryBufferMemory(\n", | ||
| " max_token_limit=64,\n", | ||
| " llm=llm,\n", | ||
| " )\n", | ||
| "\n", | ||
| " # Conversational app puts it all together.\n", | ||
| " app = ConversationChain(\n", | ||
| " llm=llm,\n", | ||
| " memory=memory\n", | ||
| " )\n", | ||
| "\n", | ||
| " return app \n", | ||
| "\n", | ||
| "app1 = load_langchain_app()\n", | ||
| "\n", | ||
| "tru_app1 = tru.Chain(\n", | ||
| " app1,\n", | ||
| " app_id='langchain_app',\n", | ||
| " initial_app_loader=load_langchain_app\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## llama_index example" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from llama_index import SimpleWebPageReader\n", | ||
| "# Be careful what you include as globals to be used by the loader function as it\n", | ||
| "# will have to be serialized. We enforce a size limit which prohibits large\n", | ||
| "# objects to be included in the loader's closure.\n", | ||
| "\n", | ||
| "# This object will be serialized alongside `load_llamaindex_app` below.\n", | ||
| "documents = SimpleWebPageReader(\n", | ||
| " html_to_text=True\n", | ||
| ").load_data([\"http://paulgraham.com/worked.html\"])\n", | ||
| "\n", | ||
| "def load_llamaindex_app():\n", | ||
| " from llama_index import VectorStoreIndex\n", | ||
| " index = VectorStoreIndex.from_documents(documents) \n", | ||
| " query_engine = index.as_query_engine()\n", | ||
| "\n", | ||
| " return query_engine\n", | ||
| "\n", | ||
| "app2 = load_llamaindex_app()\n", | ||
| "tru_app2 = tru.Llama(\n", | ||
| " app2,\n", | ||
| " app_id=\"llamaindex_app\",\n", | ||
| " initial_app_loader=load_llamaindex_app\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## basic app example" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from trulens_eval.tru_basic_app import TruWrapperApp\n", | ||
| "\n", | ||
| "def load_basic_app():\n", | ||
| " def custom_application(prompt: str) -> str:\n", | ||
| " return f\"a useful response to {prompt}\"\n", | ||
| " \n", | ||
| " return TruWrapperApp(custom_application)\n", | ||
| "\n", | ||
| "app3 = load_basic_app()\n", | ||
| "\n", | ||
| "tru_app3 = tru.Basic(\n", | ||
| " app3,\n", | ||
| " app_id=\"basic_app\",\n", | ||
| " initial_app_loader=load_basic_app\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## custom app example" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from examples.expositional.end2end_apps.custom_app.custom_app import CustomApp # our custom app\n", | ||
| "\n", | ||
| "# Create custom app:\n", | ||
| "def load_custom_app():\n", | ||
| " app = CustomApp()\n", | ||
| " return app\n", | ||
| "\n", | ||
| "app4 = load_custom_app()\n", | ||
| "\n", | ||
| "# Create trulens wrapper:\n", | ||
| "tru_app4 = tru.Custom(\n", | ||
| " app=app4,\n", | ||
| " app_id=\"custom_app\",\n", | ||
| " \n", | ||
| " # Make sure to specify using the bound method, bound to self=app.\n", | ||
| " main_method=app4.respond_to_query,\n", | ||
| "\n", | ||
| " initial_app_loader = load_custom_app\n", | ||
| ")\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Verification\n", | ||
| "\n", | ||
| "You can get a list of apps that include the `initial_app_loader` with the following utility method." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from trulens_eval.schema import AppDefinition\n", | ||
| "\n", | ||
| "for app_json in AppDefinition.get_loadable_apps():\n", | ||
| " print(app_json['app_id'])" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "py38_trulens", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.8.16" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.