diff --git a/README.md b/README.md index 0762a0506..e2c7fd1db 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,6 @@ > [!Tip] > 🔔**Claude3 Opus supported.** As of 04/17/2024, Bedrock only supports the `us-west-2` region. In this repository, Bedrock uses the `us-east-1` region by default. Therefore, if you plan to use it, please change the value of `bedrockRegion` before deployment. For more details, please refer [here](#deploy-using-cdk). -> [!Important] -> We'd like to hear your feedback to implement bot creation permission management feature. The plan is to grant permissions to individual users through the admin panel, but this may increase operational overhead for existing users. [Please take the survey](https://github.com/aws-samples/bedrock-claude-chat/issues/161#issuecomment-2058194533). - > [!Warning] > The current version (`v0.4.x`) has no compatibility with the previous version (~`v0.3.0`) due to changes in the DynamoDB table schema. **Please note that the UPDATE (i.e. `cdk deploy`) FROM PREVIOUS VERSION TO `v0.4.x` WILL DESTROY ALL OF THE EXISTING CONVERSATIONS.** @@ -200,15 +197,18 @@ BedrockChatStack.FrontendURL = https://xxxxx.cloudfront.net ``` ## Others + ### Configure Mistral models support + Update `enableMistral` to `true` in [cdk.json](./cdk/cdk.json), and run `cdk deploy`. + ```json ... "enableMistral": true, ``` -[!NOTE] -- This project focus on Anthropic Claude models, the Mistral models are limited supported. For example, prompt examples are based on Claude models. -- This is a Mistral-only option, once you toggled to enable Mistral models, you can only use Mistral models for all the chat features, NOT both Claude and Mistral models. + +> [!Important] +> This project focus on Anthropic Claude models, the Mistral models are limited supported. For example, prompt examples are based on Claude models. This is a Mistral-only option, once you toggled to enable Mistral models, you can only use Mistral models for all the chat features, NOT both Claude and Mistral models. ### Configure text generation diff --git a/docs/ADMINISTRATOR.md b/docs/ADMINISTRATOR.md index aa82060d3..a425fcd5f 100644 --- a/docs/ADMINISTRATOR.md +++ b/docs/ADMINISTRATOR.md @@ -1,16 +1,18 @@ -# Administrator dashboard +# Administrator features -The administrator dashboard is a vital tool as it provides essential insights into custom bot usage and user behavior. Without the functionality, it would be challenging for administrators to understand which custom bots are popular, why they are popular, and who is using them. This information is crucial for optimizing instruction prompts, customizing RAG data sources, and identifying heavy users who might will be an influencer. +The administrator features are a vital tool as it provides essential insights into custom bot usage and user behavior. Without the functionality, it would be challenging for administrators to understand which custom bots are popular, why they are popular, and who is using them. This information is crucial for optimizing instruction prompts, customizing RAG data sources, and identifying heavy users who might will be an influencer. -### Feedback loop +## Feedback loop The output from LLM may not always meet the user's expectations. Sometimes it fails to satisfy the user's needs. To effectively "integrate" LLMs into business operations and daily life, implementing a feedback loop is essential. Bedrock Claude Chat is equipped with a feedback feature designed to enable users to analyze why dissatisfaction arose. Based on the analysis results, users can adjust the prompts, RAG data sources, and parameters accordingly. ![](./imgs/feedback_loop.png) -![](./imgs/feedback.png) +![](./imgs/feedback-using-claude-chat.png) -## Features +Data analysts can access to conversation logs using [Amazon Athena](https://aws.amazon.com/jp/athena/). If they want to analyze the data by [Jupyter Notebook](https://jupyter.org/), [this notebook example](./notebooks/feedback_analysis_example.ipynb) can be a reference. + +## Administrator dashboard Currently provides a basic overview of chatbot and user usage, focusing on aggregating data for each bot and user over specified time periods and sorting the results by usage fees. diff --git a/docs/README_ja.md b/docs/README_ja.md index 9823c8af1..44138c950 100644 --- a/docs/README_ja.md +++ b/docs/README_ja.md @@ -3,9 +3,6 @@ > [!Tip] > 🔔**Claude3 Opus をサポートしました。** 2024/04/17 現在、Bedrock は`us-west-2`のみサポートしています。このリポジトリでは Bedrock はデフォルトで`us-east-1`リージョンを利用します。このため、ご利用される場合はデプロイ前に`bedrockRegion`の値を変更してください。詳細は[こちら](#deploy-using-cdk) -> [!Important] -> ボット作成権限の管理機能実装について、ご意見をお聞かせください。管理者パネルから個別にユーザーに権限を付与する予定ですが、既存ユーザーの皆さんの運用オーバーヘッドが大幅に増加する可能性があります。[アンケートにご協力ください](https://github.com/aws-samples/bedrock-claude-chat/issues/161#issuecomment-2058194533)。 - > [!Warning] > 現在のバージョン(v0.4.x)は、DynamoDB テーブルスキーマの変更のため、過去バージョン(~v0.3.0)とは互換性がありません。**以前のバージョンから v0.4.x へアップデートすると、既存の対話記録は全て破棄されますので注意が必要です。** @@ -189,6 +186,18 @@ BedrockChatStack.FrontendURL = https://xxxxx.cloudfront.net ## その他 +### Mistral を利用する + +cdk.json 内の`enableMistral`を`true`に更新し、`cdk deploy`を実行します。 + +```json +... + "enableMistral": true, +``` + +> [!Important] +> このプロジェクトは Anthropic の Claude モデルを中心としており、Mistral モデルはサポートが限定的です。例えば、プロンプトの例は Claude モデルを基準としています。これは Mistral モデル専用のオプションです。一度 Mistral モデルを有効にすると、すべてのチャット機能で Mistral モデルのみを使用できます。Claude モデルと Mistral モデルの両方を使用することはできません。 + ### テキスト生成パラメータの設定 [config.py](../backend/app/config.py)を編集後、`cdk deploy`を実行してください。 diff --git a/docs/imgs/feedback-using-claude-chat.png b/docs/imgs/feedback-using-claude-chat.png new file mode 100644 index 000000000..257bd50ef Binary files /dev/null and b/docs/imgs/feedback-using-claude-chat.png differ diff --git a/docs/imgs/feedback_loop.png b/docs/imgs/feedback_loop.png index 3d0caa78d..6ef61d618 100644 Binary files a/docs/imgs/feedback_loop.png and b/docs/imgs/feedback_loop.png differ diff --git a/docs/notebooks/feedback_analysis_example.ipynb b/docs/notebooks/feedback_analysis_example.ipynb new file mode 100644 index 000000000..77bf167e9 --- /dev/null +++ b/docs/notebooks/feedback_analysis_example.ipynb @@ -0,0 +1,145 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "aaceb216-d941-44a2-acef-cf11ea2df53f", + "metadata": {}, + "source": [ + "# Feedback Analysis Example\n", + "## Prerequisites\n", + "Download conversation log using Athena. Following is an example query to get conversation with user's feedback:\n", + "\n", + "```sql\n", + "SELECT \n", + " d.newimage.PK.S AS UserId,\n", + " d.newimage.SK.S AS ConvId,\n", + " d.newimage.TotalPrice.N AS TotalPrice,\n", + " d.newimage.MessageMap.S AS MessageMap\n", + "FROM \n", + " bedrockchatstack_usage_analysis.ddb_export d\n", + "WHERE \n", + " d.datehour BETWEEN '2024/04/01/00' AND '2024/04/29/23'\n", + " AND REGEXP_LIKE(d.newimage.MessageMap.S, '\"feedback\":((?!\\snull).)')\n", + "ORDER BY\n", + " d.datehour DESC\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4897c330-b12d-4bfe-88c9-6c770d98649f", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import json" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "47bc6a80-bbdf-4ead-9693-d74aa3860b40", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "from IPython import display\n", + "\n", + "def extract_conversation(data):\n", + " # Extract conversation to display.\n", + " conversation = []\n", + "\n", + " for key, value in data.items():\n", + " if key not in (\"system\", \"instruction\"):\n", + " # print(f\"key: {key}\")\n", + " # print(f\"value: {value}\")\n", + " role = value['role']\n", + " feedback = value.get('feedback')\n", + " used_chunks = value.get('used_chunks')\n", + " content = value['content'][0]['body']\n", + "\n", + " content = content.replace('\\\\n', '\\n')\n", + "\n", + " conversation.append(f\"### {role.capitalize()}:\\n\\n{content.strip()}\")\n", + " if feedback:\n", + " category = feedback.get('category')\n", + " comment = feedback.get('comment')\n", + " conversation.append(f\"### feedback:\\n\\ncategory: {category}\\n\\ncomment: {comment}\")\n", + " \n", + " if used_chunks:\n", + " conversation.append(f\"### used_chunks:\\n\\n{used_chunks}\")\n", + "\n", + " return \"\\n\\n\".join(conversation)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25388648-cc07-47d5-a214-245320c97e1c", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Enter downloaded csv file path from athena\n", + "filepath = './xxx.csv'\n", + "df = pd.read_csv(filepath)\n", + "df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7a6fc548-3d1c-4b12-947a-1dd5c6468c75", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Display conversation with feedback and used RAG chunks\n", + "display.Markdown(extract_conversation(json.loads(df[\"MessageMap\"].iloc[1])))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "65d7b127-091c-4afb-97c2-d4ae7552b3a6", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8f9d7cbb-746a-4ce6-a398-7eb292e988cf", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "conda_python3", + "language": "python", + "name": "conda_python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}