\\nDive into building GenAI applications!\\nThis repository contains examples, applications, starter code, & tutorials to help you kickstart your GenAI projects.\\n\\n- These are built using LanceDB, a free, open-source, serverless vectorDB that **requires no setup**. \\n- It **integrates into python data ecosystem** so you can simply start using these in your existing data pipelines in pandas, arrow, pydantic etc.\\n- LanceDB has **native Typescript SDK** using which you can **run vector search** in serverless functions!\\n\\n
\\nJoin our community for support - Discord •\\nTwitter\\n\\n---\\n\\nThis repository is divided into 3 sections:\\n- [Examples](#examples) - Get right into the code with minimal introduction, aimed at getting you from an idea to PoC within minutes!\\n- [Applications](#projects--applications) - Ready to use Python and web apps using applied LLMs, VectorDB and GenAI tools\\n- [Tutorials](#tutorials) - A curated list of tutorials, blogs, Colabs and courses to get you started with GenAI in greater depth.'" + "cell_type": "markdown", + "source": [ + "### Node Parser - [Simple File](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#simplefilenodeparser)\n", + "\n", + "Covering all the files intelligently" ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "HtW5uzownkVP" } - }, - "metadata": {}, - "execution_count": 10 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "## Chunking" - ], - "metadata": { - "id": "gCFoPc1PZFI5" - } - }, - { - "cell_type": "code", - "source": [ - "# Download for running Code Splitting\n", - "!wget https://raw.githubusercontent.com/lancedb/vectordb-recipes/main/applications/talk-with-podcast/app.py" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" }, - "id": "rVkeWuwvDwu-", - "outputId": "1ddb950a-0c0f-4be4-88fd-3029d53e6640" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stdout", - "text": [ - "--2024-04-15 10:22:58-- https://raw.githubusercontent.com/lancedb/vectordb-recipes/main/applications/talk-with-podcast/app.py\n", - "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...\n", - "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n", - "HTTP request sent, awaiting response... 200 OK\n", - "Length: 1582 (1.5K) [text/plain]\n", - "Saving to: ‘app.py’\n", - "\n", - "\rapp.py 0%[ ] 0 --.-KB/s \rapp.py 100%[===================>] 1.54K --.-KB/s in 0s \n", - "\n", - "2024-04-15 10:22:58 (12.1 MB/s) - ‘app.py’ saved [1582/1582]\n", - "\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Code Splitting" - ], - "metadata": { - "id": "spibLOthoCsK" - } - }, - { - "cell_type": "code", - "source": [ - "# Code Splitting\n", - "\n", - "from llama_index.core.node_parser import CodeSplitter\n", - "\n", - "documents = FlatReader().load_data(Path(\"app.py\"))\n", - "splitter = CodeSplitter(\n", - " language=\"python\",\n", - " chunk_lines=40, # lines per chunk\n", - " chunk_lines_overlap=15, # lines overlap between chunks\n", - " max_chars=1500, # max chars per chunk\n", - ")\n", - "nodes = splitter.get_nodes_from_documents(documents)\n", - "nodes[0].text" - ], - "metadata": { - "id": "IDoDzDeiYqpL", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 140 + "cell_type": "code", + "source": [ + "# Simple File\n", + "from llama_index.core.node_parser import SimpleFileNodeParser\n", + "from llama_index.readers.file import FlatReader\n", + "from pathlib import Path\n", + "\n", + "md_docs = FlatReader().load_data(Path(\"README.md\"))\n", + "\n", + "parser = SimpleFileNodeParser()\n", + "\n", + "# Additionally, you can augment this with a text-based parser to accurately handle text length\n", + "md_nodes = parser.get_nodes_from_documents(md_docs)\n", + "md_nodes[0].text" + ], + "metadata": { + "id": "GqWdmhBdWrB4", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 105 + }, + "outputId": "172b7873-06cc-4a71-fd5b-acdcd24aeb0d" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'VectorDB-recipes\\n
\\nDive into building GenAI applications!\\nThis repository contains examples, applications, starter code, & tutorials to help you kickstart your GenAI projects.\\n\\n- These are built using LanceDB, a free, open-source, serverless vectorDB that **requires no setup**. \\n- It **integrates into python data ecosystem** so you can simply start using these in your existing data pipelines in pandas, arrow, pydantic etc.\\n- LanceDB has **native Typescript SDK** using which you can **run vector search** in serverless functions!\\n\\n
\\nJoin our community for support - Discord •\\nTwitter\\n\\n---\\n\\nThis repository is divided into 3 sections:\\n- [Examples](#examples) - Get right into the code with minimal introduction, aimed at getting you from an idea to PoC within minutes!\\n- [Applications](#projects--applications) - Ready to use Python and web apps using applied LLMs, VectorDB and GenAI tools\\n- [Tutorials](#tutorials) - A curated list of tutorials, blogs, Colabs and courses to get you started with GenAI in greater depth.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 4 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Node Parser - [HTML](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#htmlnodeparser)\n", + "\n", + "This node parser uses beautifulsoup to parse raw HTML.\n", + "\n", + "By default, it will parse a select subset of HTML tags, but you can override this.\n", + "\n", + "The default tags are: [\"p\", \"h1\", \"h2\", \"h3\", \"h4\", \"h5\", \"h6\", \"li\", \"b\", \"i\", \"u\", \"section\"]" + ], + "metadata": { + "id": "-au7BAS2nvBC" + } }, - "outputId": "5ac4578a-c5de-4060-cf9c-420a9078652b" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stderr", - "text": [ - "/usr/local/lib/python3.10/dist-packages/tree_sitter/__init__.py:36: FutureWarning: Language(path, name) is deprecated. Use Language(ptr, name) instead.\n", - " warn(\"{} is deprecated. Use {} instead.\".format(old, new), FutureWarning)\n" - ] + "cell_type": "code", + "source": [ + "# HTML\n", + "\n", + "import requests\n", + "from llama_index.core import Document\n", + "from llama_index.core.node_parser import HTMLNodeParser\n", + "\n", + "# URL of the website to fetch HTML from\n", + "url = \"https://www.utoronto.ca/\"\n", + "\n", + "# Send a GET request to the URL\n", + "response = requests.get(url)\n", + "print(response)\n", + "\n", + "# Check if the request was successful (status code 200)\n", + "if response.status_code == 200:\n", + " # Extract the HTML content from the response\n", + " html_doc = response.text\n", + " document = Document(id_=url, text=html_doc)\n", + "\n", + " parser = HTMLNodeParser(tags=[\"p\", \"h1\"])\n", + " nodes = parser.get_nodes_from_documents([document])\n", + " print(nodes)\n", + "else:\n", + " # Print an error message if the request was unsuccessful\n", + " print(\"Failed to fetch HTML content:\", response.status_code)" + ], + "metadata": { + "id": "Zhe7xYJtXw4l", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7f32b9e5-225e-4a3e-b9a0-9a6287615f5a" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "
\\nDive into building GenAI applications!\\nThis repository contains examples, applications, starter code, & tutorials to help you kickstart your GenAI projects.\\n\\n- These are built using LanceDB, a free, open-source, serverless vectorDB that **requires no setup**. \\n- It **integrates into python data ecosystem** so you can simply start using these in your existing data pipelines in pandas, arrow, pydantic etc.\\n- LanceDB has **native Typescript SDK** using which you can **run vector search** in serverless functions!\\n\\n
\\nJoin our community for support - Discord •\\nTwitter\\n\\n---\\n\\nThis repository is divided into 3 sections:\\n- [Examples](#examples) - Get right into the code with minimal introduction, aimed at getting you from an idea to PoC within minutes!\\n- [Applications](#projects--applications) - Ready to use Python and web apps using applied LLMs, VectorDB and GenAI tools\\n- [Tutorials](#tutorials) - A curated list of tutorials, blogs, Colabs and courses to get you started with GenAI in greater depth.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 10 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Text-Splitters\n", + "\n", + "Download a `.py` file for some other chunking methods" ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "gCFoPc1PZFI5" } - }, - "metadata": {}, - "execution_count": 16 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Node Parser - Semantic Splitting" - ], - "metadata": { - "id": "zj45BKLMoRgp" - } - }, - { - "cell_type": "code", - "source": [ - "# SemanticSplitterNodeParser\n", - "\n", - "from llama_index.core.node_parser import SemanticSplitterNodeParser\n", - "from llama_index.embeddings.openai import OpenAIEmbedding\n", - "import os\n", - "\n", - "# Add OpenAI API key as environment variable\n", - "os.environ[\"OPENAI_API_KEY\"] = \"sk-****\"\n", - "\n", - "embed_model = OpenAIEmbedding()\n", - "splitter = SemanticSplitterNodeParser(\n", - " buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model\n", - ")\n", - "\n", - "nodes = splitter.get_nodes_from_documents(documents)\n", - "nodes[0].text" - ], - "metadata": { - "id": "wAp7BU25ZdRt", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 53 }, - "outputId": "3f51c53b-7617-4c67-c247-125f7a6b84be" - }, - "execution_count": null, - "outputs": [ { - "output_type": "execute_result", - "data": { - "text/plain": [ - "'Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. '" + "cell_type": "code", + "source": [ + "# Download for running Code Splitting\n", + "!wget https://raw.githubusercontent.com/lancedb/vectordb-recipes/main/applications/talk-with-podcast/app.py" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "rVkeWuwvDwu-", + "outputId": "1ddb950a-0c0f-4be4-88fd-3029d53e6640" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "--2024-04-15 10:22:58-- https://raw.githubusercontent.com/lancedb/vectordb-recipes/main/applications/talk-with-podcast/app.py\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 1582 (1.5K) [text/plain]\n", + "Saving to: ‘app.py’\n", + "\n", + "\rapp.py 0%[ ] 0 --.-KB/s \rapp.py 100%[===================>] 1.54K --.-KB/s in 0s \n", + "\n", + "2024-04-15 10:22:58 (12.1 MB/s) - ‘app.py’ saved [1582/1582]\n", + "\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### [Code Splitting](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#codesplitter)\n", + "\n", + "Splits raw code-text based on the language it is written in.\n", + "\n", + "Check the full list of [supported languages here](https://github.com/grantjenks/py-tree-sitter-languages#license)." ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "spibLOthoCsK" } - }, - "metadata": {}, - "execution_count": 17 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Token Text Splitting" - ], - "metadata": { - "id": "vH9xni1SoWYE" - } - }, - { - "cell_type": "code", - "source": [ - "# TokenTextSplitting\n", - "\n", - "from llama_index.core.node_parser import TokenTextSplitter\n", - "\n", - "splitter = TokenTextSplitter(\n", - " chunk_size=254,\n", - " chunk_overlap=20,\n", - " separator=\" \",\n", - ")\n", - "nodes = splitter.get_nodes_from_documents(documents)\n", - "nodes[0].text" - ], - "metadata": { - "id": "9G61og__Ziec", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 105 }, - "outputId": "5b11b5f9-94d1-4b6d-a7d5-a58025f58f2a" - }, - "execution_count": null, - "outputs": [ { - "output_type": "execute_result", - "data": { - "text/plain": [ - "\"Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\\n\\nIt's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\\n\\nAgain, we are tested. And again, we must answer history's call.\\n\\nOne year ago, I took office amid two wars, an economy\"" + "cell_type": "code", + "source": [ + "# Code Splitting\n", + "\n", + "from llama_index.core.node_parser import CodeSplitter\n", + "\n", + "documents = FlatReader().load_data(Path(\"app.py\"))\n", + "splitter = CodeSplitter(\n", + " language=\"python\",\n", + " chunk_lines=40, # lines per chunk\n", + " chunk_lines_overlap=15, # lines overlap between chunks\n", + " max_chars=1500, # max chars per chunk\n", + ")\n", + "nodes = splitter.get_nodes_from_documents(documents)\n", + "nodes[0].text" ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "IDoDzDeiYqpL", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 140 + }, + "outputId": "5ac4578a-c5de-4060-cf9c-420a9078652b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.10/dist-packages/tree_sitter/__init__.py:36: FutureWarning: Language(path, name) is deprecated. Use Language(ptr, name) instead.\n", + " warn(\"{} is deprecated. Use {} instead.\".format(old, new), FutureWarning)\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'from youtube_podcast_download import podcast_audio_retreival\\nfrom transcribe_podcast import transcribe\\nfrom chat_retreival import retrieverSetup, chat\\nfrom langroid_utils import configure, agent\\n\\nimport os\\nimport glob\\nimport json\\nimport streamlit as st\\n\\nOPENAI_KEY = os.environ[\"OPENAI_API_KEY\"]\\n\\n\\n@st.cache_resource\\ndef video_data_retreival(framework):\\n f = open(\"output.json\")\\n data = json.load(f)\\n\\n # setting up reteriver\\n if framework == \"Langchain\":\\n qa = retrieverSetup(data[\"text\"], OPENAI_KEY)\\n return qa\\n elif framework == \"Langroid\":\\n langroid_file = open(\"langroid_doc.txt\", \"w\") # write mode\\n langroid_file.write(data[\"text\"])\\n cfg = configure(\"langroid_doc.txt\")\\n return cfg\\n\\n\\nst.header(\"Talk with Youtube Podcasts\", divider=\"rainbow\")\\n\\nurl = st.text_input(\"Youtube Link\")\\nframework = st.radio(\\n \"**Select Framework 👇**\",\\n [\"Langchain\", \"Langroid\"],\\n key=\"Langchain\",\\n)\\n\\nif url:\\n st.video(url)\\n # Podcast Audio Retreival from Youtube\\n podcast_audio_retreival(url)\\n\\n # Trascribing podcast audio\\n filename = glob.glob(\"*.mp3\")[0]\\n transcribe(filename)\\n\\n st.markdown(f\"##### `{framework}` Framework Selected for talking with Podcast\")\\n # Chat Agent getting ready\\n qa = video_data_retreival(framework)\\n\\n\\nprompt = st.chat_input(\"Talk with Podcast\")\\n\\ni'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 13 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### [Sentence Splitting](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#sentencesplitter)\n", + "The SentenceSplitter attempts to split text while respecting the boundaries of sentences." + ], + "metadata": { + "id": "On40RuqBoGNL" } - }, - "metadata": {}, - "execution_count": 18 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "## Relation based Node Parser" - ], - "metadata": { - "id": "rpbqXxeaawOt" - } - }, - { - "cell_type": "markdown", - "source": [ - "### Node Parser - Hierarchical" - ], - "metadata": { - "id": "z_HuwzzAoabc" - } - }, - { - "cell_type": "code", - "source": [ - "# HierarchicalNodeParser\n", - "\n", - "from llama_index.core.node_parser import HierarchicalNodeParser\n", - "\n", - "node_parser = HierarchicalNodeParser.from_defaults(chunk_sizes=[512, 254, 128])\n", - "\n", - "nodes = node_parser.get_nodes_from_documents(documents)\n", - "nodes[0].text" - ], - "metadata": { - "id": "qwZFEDlaZpKT", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 105 }, - "outputId": "12888840-daf5-45f5-f934-e253b6036621" - }, - "execution_count": null, - "outputs": [ { - "output_type": "execute_result", - "data": { - "text/plain": [ - "\"Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\\n\\nIt's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\\n\\nAgain, we are tested. And again, we must answer history's call.\\n\\nOne year ago, I took office amid two wars, an economy rocked by severe recession, a financial system on the verge of collapse and a government deeply in debt. Experts from across the political spectrum warned that if we did not act, we might face a second depression. So we acted immediately and aggressively. And one year later, the worst of the storm has passed.\\n\\nBut the devastation remains. One in 10 Americans still cannot find work. Many businesses have shuttered. Home values have declined. Small towns and rural communities have been hit especially hard. For those who had already known poverty, life has become that much harder.\\n\\nThis recession has also compounded the burdens that America's families have been dealing with for decades -- the burden of working harder and longer for less, of being unable to save enough to retire or help kids with college.\\n\\nSo I know the anxieties that are out there right now. They're not new. These struggles are the reason I ran for president. These struggles are what I've witnessed for years in places like Elkhart, Ind., and Galesburg, Ill. I hear about them in the letters that I read each night.\"" + "cell_type": "code", + "source": [ + "# Sentence Splitting\n", + "\n", + "from llama_index.core.node_parser import SentenceSplitter\n", + "\n", + "documents = FlatReader().load_data(Path(\"state_of_the_union.txt\"))\n", + "splitter = SentenceSplitter(\n", + " chunk_size=254,\n", + " chunk_overlap=20,\n", + ")\n", + "nodes = splitter.get_nodes_from_documents(documents)\n", + "nodes[0].text" + ], + "metadata": { + "id": "iNKuiCNrZOHl", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 105 + }, + "outputId": "47064bf4-4079-42df-83b6-d519ba92a135" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "\"Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\\n\\nIt's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\\n\\nAgain, we are tested. And again, we must answer history's call.\"" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Node Parser - [Sentence Window](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#sentencewindownodeparser)\n", + "\n", + "The SentenceWindowNodeParser is similar to other node parsers, except that it splits all documents into individual sentences. The resulting nodes also contain the surrounding \"window\" of sentences around each node in the metadata. Note that this metadata will not be visible to the LLM or embedding model.\n", + "\n", + "This is most useful for generating embeddings that have a very specific scope. Then, combined with a MetadataReplacementNodePostProcessor, you can replace the sentence with it's surrounding context before sending the node to the LLM.\n", + "\n", + "An example of setting up the parser with default settings is below. In practice, you would usually only want to adjust the window size of sentences." ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "h5zc7_YmoJHP" } - }, - "metadata": {}, - "execution_count": 19 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "# Langchain Text Chunking Strategies" - ], - "metadata": { - "id": "64yUhjV_a9dk" - } - }, - { - "cell_type": "code", - "source": [ - "!pip install -qU langchain-text-splitters\n", - "!pip install requests" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" }, - "id": "8OijyBvqbCLC", - "outputId": "8c5d46c4-0435-4f56-fe50-34c28d7846fd" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stdout", - "text": [ - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.5/287.5 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m113.0/113.0 kB\u001b[0m \u001b[31m11.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", - "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m144.8/144.8 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", - "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.31.0)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests) (3.3.2)\n", - "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests) (3.6)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests) (2.0.7)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests) (2024.2.2)\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Text Splitting - Character" - ], - "metadata": { - "id": "QVKmz3Rvok9Y" - } - }, - { - "cell_type": "code", - "source": [ - "# Split with Character\n", - "\n", - "with open(\"state_of_the_union.txt\") as f:\n", - " state_of_the_union = f.read()\n", - "\n", - "\n", - "from langchain_text_splitters import CharacterTextSplitter\n", - "\n", - "text_splitter = CharacterTextSplitter(\n", - " separator=\"\\n\\n\",\n", - " chunk_size=1000,\n", - " chunk_overlap=200,\n", - " length_function=len,\n", - " is_separator_regex=False,\n", - ")\n", - "\n", - "texts = text_splitter.create_documents([state_of_the_union])\n", - "print(texts[0].page_content)" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" + "cell_type": "code", + "source": [ + "# SentenceWindowNodeParser\n", + "\n", + "import nltk\n", + "from llama_index.core.node_parser import SentenceWindowNodeParser\n", + "\n", + "node_parser = SentenceWindowNodeParser.from_defaults(\n", + " window_size=3,\n", + " window_metadata_key=\"window\",\n", + " original_text_metadata_key=\"original_sentence\",\n", + ")\n", + "nodes = node_parser.get_nodes_from_documents(documents)\n", + "nodes[0].text" + ], + "metadata": { + "id": "76tbzUrMZRFF", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 53 + }, + "outputId": "7f57513d-da7b-45f9-96c0-5698e06f1562" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. '" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 16 + } + ] }, - "id": "EdjiGEitI4La", - "outputId": "29bb4bfd-e198-4902-e7c0-40c6df7d488b" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stderr", - "text": [ - "WARNING:langchain_text_splitters.base:Created a chunk of size 1163, which is longer than the specified 1000\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 1015, which is longer than the specified 1000\n" - ] + "cell_type": "markdown", + "source": [ + "### Node Parser - [Semantic Splitting](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#semanticsplitternodeparser)\n", + "\"Semantic chunking\" is a new concept proposed Greg Kamradt in his video tutorial on 5 levels of embedding chunking: https://youtu.be/8OJC21T2SL4?t=1933.\n", + "\n", + "Instead of chunking text with a fixed chunk size, the semantic splitter adaptively picks the breakpoint in-between sentences using embedding similarity. This ensures that a \"chunk\" contains sentences that are semantically related to each other." + ], + "metadata": { + "id": "zj45BKLMoRgp" + } }, { - "output_type": "stream", - "name": "stdout", - "text": [ - "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\n", - "\n", - "Our Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Text Splitting - Recursive Character" - ], - "metadata": { - "id": "ivNYVKPZowKh" - } - }, - { - "cell_type": "code", - "source": [ - "# Recursive Split Character\n", - "\n", - "# This is a long document we can split up.\n", - "with open(\"state_of_the_union.txt\") as f:\n", - " state_of_the_union = f.read()\n", - "\n", - "from langchain_text_splitters import RecursiveCharacterTextSplitter\n", - "\n", - "text_splitter = RecursiveCharacterTextSplitter(\n", - " # Set a really small chunk size, just to show.\n", - " chunk_size=1000,\n", - " chunk_overlap=100,\n", - " length_function=len,\n", - " is_separator_regex=False,\n", - ")\n", - "\n", - "texts = text_splitter.create_documents([state_of_the_union])\n", - "print(\"Chunk 2: \", texts[1].page_content)\n", - "print(\"Chunk 3: \", texts[2].page_content)" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" + "cell_type": "code", + "source": [ + "# SemanticSplitterNodeParser\n", + "\n", + "from llama_index.core.node_parser import SemanticSplitterNodeParser\n", + "from llama_index.embeddings.openai import OpenAIEmbedding\n", + "import os\n", + "\n", + "# Add OpenAI API key as environment variable\n", + "os.environ[\"OPENAI_API_KEY\"] = \"sk-****\"\n", + "\n", + "embed_model = OpenAIEmbedding()\n", + "splitter = SemanticSplitterNodeParser(\n", + " buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model\n", + ")\n", + "\n", + "nodes = splitter.get_nodes_from_documents(documents)\n", + "nodes[0].text" + ], + "metadata": { + "id": "wAp7BU25ZdRt", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 53 + }, + "outputId": "3f51c53b-7617-4c67-c247-125f7a6b84be" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. '" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 17 + } + ] }, - "id": "9X6_duxwN3nI", - "outputId": "d6ce1302-1a9a-4887-9505-1c206390ab2f" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stdout", - "text": [ - "Chunk 2: It's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\n", - "\n", - "Again, we are tested. And again, we must answer history's call.\n", - "Chunk 3: Again, we are tested. And again, we must answer history's call.\n", - "\n", - "One year ago, I took office amid two wars, an economy rocked by severe recession, a financial system on the verge of collapse and a government deeply in debt. Experts from across the political spectrum warned that if we did not act, we might face a second depression. So we acted immediately and aggressively. And one year later, the worst of the storm has passed.\n", - "\n", - "But the devastation remains. One in 10 Americans still cannot find work. Many businesses have shuttered. Home values have declined. Small towns and rural communities have been hit especially hard. For those who had already known poverty, life has become that much harder.\n", - "\n", - "This recession has also compounded the burdens that America's families have been dealing with for decades -- the burden of working harder and longer for less, of being unable to save enough to retire or help kids with college.\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Text Splitting - HTML Header" - ], - "metadata": { - "id": "I1nKMkm4o1Ft" - } - }, - { - "cell_type": "code", - "source": [ - "# Split with HTML Tags\n", - "\n", - "from langchain_text_splitters import HTMLHeaderTextSplitter\n", - "import requests\n", - "\n", - "# URL of the website to fetch HTML from\n", - "url = \"https://www.utoronto.ca/\"\n", - "\n", - "# Send a GET request to the URL\n", - "response = requests.get(url)\n", - "if response.status_code == 200:\n", - " html_doc = response.text\n", - "\n", - "headers_to_split_on = [\n", - " (\"h1\", \"Header 1\"),\n", - " (\"h2\", \"Header 2\"),\n", - " (\"h3\", \"Header 3\"),\n", - "]\n", - "\n", - "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)\n", - "html_header_splits = html_splitter.split_text(html_doc)\n", - "html_header_splits[0].page_content" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 35 + "cell_type": "markdown", + "source": [ + "### [Token Text Splitting](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#tokentextsplitter)\n", + "\n", + "The TokenTextSplitter attempts to split to a consistent chunk size according to raw token counts." + ], + "metadata": { + "id": "vH9xni1SoWYE" + } + }, + { + "cell_type": "code", + "source": [ + "# TokenTextSplitting\n", + "\n", + "from llama_index.core.node_parser import TokenTextSplitter\n", + "\n", + "splitter = TokenTextSplitter(\n", + " chunk_size=254,\n", + " chunk_overlap=20,\n", + " separator=\" \",\n", + ")\n", + "nodes = splitter.get_nodes_from_documents(documents)\n", + "nodes[0].text" + ], + "metadata": { + "id": "9G61og__Ziec", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 105 + }, + "outputId": "5b11b5f9-94d1-4b6d-a7d5-a58025f58f2a" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "\"Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\\n\\nIt's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\\n\\nAgain, we are tested. And again, we must answer history's call.\\n\\nOne year ago, I took office amid two wars, an economy\"" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 18 + } + ] }, - "id": "671M7BEVJ5zL", - "outputId": "d6d5df83-98f5-42e5-c50a-de7455a46b93" - }, - "execution_count": null, - "outputs": [ { - "output_type": "execute_result", - "data": { - "text/plain": [ - "'Welcome to University of Toronto \\nMain menu tools'" + "cell_type": "markdown", + "source": [ + "## Relation based Node Parser" ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "rpbqXxeaawOt" } - }, - "metadata": {}, - "execution_count": 29 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Text Splitting - Code" - ], - "metadata": { - "id": "y8utGi0No6tr" - } - }, - { - "cell_type": "code", - "source": [ - "# Code Splitting\n", - "\n", - "from langchain_text_splitters import Language, RecursiveCharacterTextSplitter\n", - "\n", - "\n", - "with open(\"app.py\") as f:\n", - " code = f.read()\n", - "\n", - "python_splitter = RecursiveCharacterTextSplitter.from_language(\n", - " language=Language.PYTHON, chunk_size=100, chunk_overlap=0\n", - ")\n", - "python_docs = python_splitter.create_documents([code])\n", - "python_docs[0].page_content" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 35 }, - "id": "9nfbAYj1KGQL", - "outputId": "92edafe3-7d0c-4e20-90ea-5111c565b232" - }, - "execution_count": null, - "outputs": [ { - "output_type": "execute_result", - "data": { - "text/plain": [ - "'from youtube_podcast_download import podcast_audio_retreival'" + "cell_type": "markdown", + "source": [ + "### Node Parser - [Hierarchical](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#hierarchicalnodeparser)\n", + "This node parser will chunk nodes into hierarchical nodes. This means a single input will be chunked into several hierarchies of chunk sizes, with each node containing a reference to it's parent node.\n", + "\n", + "When combined with the AutoMergingRetriever, this enables us to automatically replace retrieved nodes with their parents when a majority of children are retrieved. This process provides the LLM with more complete context for response synthesis." ], - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" + "metadata": { + "id": "z_HuwzzAoabc" } - }, - "metadata": {}, - "execution_count": 33 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Text Splitting - Recursive JSON" - ], - "metadata": { - "id": "tePMloUspEcX" - } - }, - { - "cell_type": "code", - "source": [ - "# Recursive Split Json\n", - "\n", - "from langchain_text_splitters import RecursiveJsonSplitter\n", - "import json\n", - "import requests\n", - "\n", - "json_data = requests.get(\"https://api.smith.langchain.com/openapi.json\").json()\n", - "\n", - "splitter = RecursiveJsonSplitter(max_chunk_size=300)\n", - "json_chunks = splitter.split_json(json_data=json_data)\n", - "json_chunks[0]" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" }, - "id": "8bW_6wkmMAoR", - "outputId": "73dadc8f-30bc-491f-c3f5-a95e75486971" - }, - "execution_count": null, - "outputs": [ { - "output_type": "execute_result", - "data": { - "text/plain": [ - "{'openapi': '3.1.0',\n", - " 'info': {'title': 'LangSmith', 'version': '0.1.0'},\n", - " 'servers': [{'url': 'https://api.smith.langchain.com',\n", - " 'description': 'LangSmith API endpoint.'}]}" + "cell_type": "code", + "source": [ + "# HierarchicalNodeParser\n", + "\n", + "from llama_index.core.node_parser import HierarchicalNodeParser\n", + "\n", + "node_parser = HierarchicalNodeParser.from_defaults(chunk_sizes=[512, 254, 128])\n", + "\n", + "nodes = node_parser.get_nodes_from_documents(documents)\n", + "nodes[0].text" + ], + "metadata": { + "id": "qwZFEDlaZpKT", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 105 + }, + "outputId": "12888840-daf5-45f5-f934-e253b6036621" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "\"Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\\n\\nOur Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\\n\\nIt's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\\n\\nAgain, we are tested. And again, we must answer history's call.\\n\\nOne year ago, I took office amid two wars, an economy rocked by severe recession, a financial system on the verge of collapse and a government deeply in debt. Experts from across the political spectrum warned that if we did not act, we might face a second depression. So we acted immediately and aggressively. And one year later, the worst of the storm has passed.\\n\\nBut the devastation remains. One in 10 Americans still cannot find work. Many businesses have shuttered. Home values have declined. Small towns and rural communities have been hit especially hard. For those who had already known poverty, life has become that much harder.\\n\\nThis recession has also compounded the burdens that America's families have been dealing with for decades -- the burden of working harder and longer for less, of being unable to save enough to retire or help kids with college.\\n\\nSo I know the anxieties that are out there right now. They're not new. These struggles are the reason I ran for president. These struggles are what I've witnessed for years in places like Elkhart, Ind., and Galesburg, Ill. I hear about them in the letters that I read each night.\"" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 19 + } ] - }, - "metadata": {}, - "execution_count": 33 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Semantic Splitting" - ], - "metadata": { - "id": "a8Rt52AepNNk" - } - }, - { - "cell_type": "code", - "source": [ - "# Semantic Chunking\n", - "\n", - "!pip install --quiet langchain_experimental langchain_openai\n", - "\n", - "import os\n", - "from langchain_experimental.text_splitter import SemanticChunker\n", - "from langchain_openai.embeddings import OpenAIEmbeddings\n", - "\n", - "# Add OpenAI API key as environment variable\n", - "os.environ[\"OPENAI_API_KEY\"] = \"sk-****\"\n", - "\n", - "with open(\"state_of_the_union.txt\") as f:\n", - " state_of_the_union = f.read()\n", - "\n", - "text_splitter = SemanticChunker(OpenAIEmbeddings())\n", - "\n", - "docs = text_splitter.create_documents([state_of_the_union])\n", - "print(docs[0].page_content)" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" }, - "id": "oHDYFAHeOjPA", - "outputId": "f7a5bbc2-c432-4370-bbf5-e529f4ff8c77" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stdout", - "text": [ - "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\n", - "\n", - "Our Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty.\n" - ] - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Splitting by Tokens" - ], - "metadata": { - "id": "dV7RMi7_pRWn" - } - }, - { - "cell_type": "code", - "source": [ - "# Splits by Tokens\n", - "\n", - "# Using Tiktoken\n", - "!pip install --upgrade --quiet tiktoken\n", - "\n", - "with open(\"state_of_the_union.txt\") as f:\n", - " state_of_the_union = f.read()\n", - "\n", - "from langchain_text_splitters import CharacterTextSplitter\n", - "\n", - "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n", - " chunk_size=100, chunk_overlap=0\n", - ")\n", - "texts = text_splitter.split_text(state_of_the_union)\n", - "\n", - "print(texts[0])" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" + "cell_type": "markdown", + "source": [ + "# Langchain Text Chunking Strategies" + ], + "metadata": { + "id": "64yUhjV_a9dk" + } + }, + { + "cell_type": "code", + "source": [ + "# install required packages and modules\n", + "!pip install -qU langchain-text-splitters\n", + "!pip install requests" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8OijyBvqbCLC", + "outputId": "8c5d46c4-0435-4f56-fe50-34c28d7846fd" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.5/287.5 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m113.0/113.0 kB\u001b[0m \u001b[31m11.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m144.8/144.8 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.31.0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests) (3.3.2)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests) (3.6)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests) (2.0.7)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests) (2024.2.2)\n" + ] + } + ] }, - "id": "7_WVw_kEQmJg", - "outputId": "93f501ed-d8a4-4350-a670-a6af25d2879d" - }, - "execution_count": null, - "outputs": [ { - "output_type": "stream", - "name": "stderr", - "text": [ - "WARNING:langchain_text_splitters.base:Created a chunk of size 123, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 104, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 109, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 106, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 129, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 111, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 118, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 132, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 231, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 177, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 112, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 130, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 116, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 184, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 139, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 112, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 151, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 203, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 138, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 123, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 213, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 134, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 130, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 125, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 139, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 111, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 130, which is longer than the specified 100\n", - "WARNING:langchain_text_splitters.base:Created a chunk of size 124, which is longer than the specified 100\n" - ] + "cell_type": "markdown", + "source": [ + "### Text Splitting - [Character](https://python.langchain.com/api_reference/text_splitters/character/langchain_text_splitters.character.CharacterTextSplitter.html#charactertextsplitter)\n", + "\n", + "Splitting text that looks at characters." + ], + "metadata": { + "id": "QVKmz3Rvok9Y" + } + }, + { + "cell_type": "code", + "source": [ + "# Split with Character\n", + "\n", + "with open(\"state_of_the_union.txt\") as f:\n", + " state_of_the_union = f.read()\n", + "\n", + "\n", + "from langchain_text_splitters import CharacterTextSplitter\n", + "\n", + "text_splitter = CharacterTextSplitter(\n", + " separator=\"\\n\\n\",\n", + " chunk_size=1000,\n", + " chunk_overlap=200,\n", + " length_function=len,\n", + " is_separator_regex=False,\n", + ")\n", + "\n", + "texts = text_splitter.create_documents([state_of_the_union])\n", + "print(texts[0].page_content)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "EdjiGEitI4La", + "outputId": "29bb4bfd-e198-4902-e7c0-40c6df7d488b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "WARNING:langchain_text_splitters.base:Created a chunk of size 1163, which is longer than the specified 1000\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 1015, which is longer than the specified 1000\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\n", + "\n", + "Our Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\n" + ] + } + ] }, { - "output_type": "stream", - "name": "stdout", - "text": [ - "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\n", - "\n", - "Our Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\n" - ] + "cell_type": "markdown", + "source": [ + "### Text Splitting - [Recursive Character](https://python.langchain.com/api_reference/text_splitters/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html)\n", + "\n", + "Splitting text by recursively look at characters.\n", + "\n", + "Recursively tries to split by different characters to find one that works.\n", + "\n" + ], + "metadata": { + "id": "ivNYVKPZowKh" + } + }, + { + "cell_type": "code", + "source": [ + "# Recursive Split Character\n", + "\n", + "# This is a long document we can split up.\n", + "with open(\"state_of_the_union.txt\") as f:\n", + " state_of_the_union = f.read()\n", + "\n", + "from langchain_text_splitters import RecursiveCharacterTextSplitter\n", + "\n", + "text_splitter = RecursiveCharacterTextSplitter(\n", + " # Set a really small chunk size, just to show.\n", + " chunk_size=1000,\n", + " chunk_overlap=100,\n", + " length_function=len,\n", + " is_separator_regex=False,\n", + ")\n", + "\n", + "texts = text_splitter.create_documents([state_of_the_union])\n", + "print(\"Chunk 2: \", texts[1].page_content)\n", + "print(\"Chunk 3: \", texts[2].page_content)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "9X6_duxwN3nI", + "outputId": "d6ce1302-1a9a-4887-9505-1c206390ab2f" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Chunk 2: It's tempting to look back on these moments and assume that our progress was inevitable, that America was always destined to succeed. But when the Union was turned back at Bull Run and the Allies first landed at Omaha Beach, victory was very much in doubt. When the market crashed on Black Tuesday and civil rights marchers were beaten on Bloody Sunday, the future was anything but certain. These were times that tested the courage of our convictions and the strength of our union. And despite all our divisions and disagreements, our hesitations and our fears, America prevailed because we chose to move forward as one nation and one people.\n", + "\n", + "Again, we are tested. And again, we must answer history's call.\n", + "Chunk 3: Again, we are tested. And again, we must answer history's call.\n", + "\n", + "One year ago, I took office amid two wars, an economy rocked by severe recession, a financial system on the verge of collapse and a government deeply in debt. Experts from across the political spectrum warned that if we did not act, we might face a second depression. So we acted immediately and aggressively. And one year later, the worst of the storm has passed.\n", + "\n", + "But the devastation remains. One in 10 Americans still cannot find work. Many businesses have shuttered. Home values have declined. Small towns and rural communities have been hit especially hard. For those who had already known poverty, life has become that much harder.\n", + "\n", + "This recession has also compounded the burdens that America's families have been dealing with for decades -- the burden of working harder and longer for less, of being unable to save enough to retire or help kids with college.\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Text Splitting - [HTML Header](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html#htmlheadertextsplitter)\n", + "\n", + "Splitting HTML files based on specified headers.\n", + "\n", + "Requires lxml package." + ], + "metadata": { + "id": "I1nKMkm4o1Ft" + } + }, + { + "cell_type": "code", + "source": [ + "# Split with HTML Tags\n", + "\n", + "from langchain_text_splitters import HTMLHeaderTextSplitter\n", + "import requests\n", + "\n", + "# URL of the website to fetch HTML from\n", + "url = \"https://www.utoronto.ca/\"\n", + "\n", + "# Send a GET request to the URL\n", + "response = requests.get(url)\n", + "if response.status_code == 200:\n", + " html_doc = response.text\n", + "\n", + "headers_to_split_on = [\n", + " (\"h1\", \"Header 1\"),\n", + " (\"h2\", \"Header 2\"),\n", + " (\"h3\", \"Header 3\"),\n", + "]\n", + "\n", + "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)\n", + "html_header_splits = html_splitter.split_text(html_doc)\n", + "html_header_splits[0].page_content" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "671M7BEVJ5zL", + "outputId": "d6d5df83-98f5-42e5-c50a-de7455a46b93" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'Welcome to University of Toronto \\nMain menu tools'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 29 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Text Splitting - Code" + ], + "metadata": { + "id": "y8utGi0No6tr" + } + }, + { + "cell_type": "code", + "source": [ + "# Code Splitting\n", + "\n", + "from langchain_text_splitters import Language, RecursiveCharacterTextSplitter\n", + "\n", + "\n", + "with open(\"app.py\") as f:\n", + " code = f.read()\n", + "\n", + "python_splitter = RecursiveCharacterTextSplitter.from_language(\n", + " language=Language.PYTHON, chunk_size=100, chunk_overlap=0\n", + ")\n", + "python_docs = python_splitter.create_documents([code])\n", + "python_docs[0].page_content" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "9nfbAYj1KGQL", + "outputId": "92edafe3-7d0c-4e20-90ea-5111c565b232" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'from youtube_podcast_download import podcast_audio_retreival'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 33 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Text Splitting - [Recursive JSON](https://python.langchain.com/api_reference/text_splitters/json/langchain_text_splitters.json.RecursiveJsonSplitter.html#recursivejsonsplitter)\n", + "\n", + "Splits JSON data into smaller, structured chunks while preserving hierarchy.\n", + "\n", + "This method splits JSON data into smaller dictionaries or JSON-formatted strings based on configurable maximum and minimum chunk sizes. It supports nested JSON structures, optionally converts lists into dictionaries for better chunking, and allows the creation of document objects for further use." + ], + "metadata": { + "id": "tePMloUspEcX" + } + }, + { + "cell_type": "code", + "source": [ + "# Recursive Split Json\n", + "\n", + "from langchain_text_splitters import RecursiveJsonSplitter\n", + "import json\n", + "import requests\n", + "\n", + "json_data = requests.get(\"https://api.smith.langchain.com/openapi.json\").json()\n", + "\n", + "splitter = RecursiveJsonSplitter(max_chunk_size=300)\n", + "json_chunks = splitter.split_json(json_data=json_data)\n", + "json_chunks[0]" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8bW_6wkmMAoR", + "outputId": "73dadc8f-30bc-491f-c3f5-a95e75486971" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "{'openapi': '3.1.0',\n", + " 'info': {'title': 'LangSmith', 'version': '0.1.0'},\n", + " 'servers': [{'url': 'https://api.smith.langchain.com',\n", + " 'description': 'LangSmith API endpoint.'}]}" + ] + }, + "metadata": {}, + "execution_count": 33 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### [Semantic Splitting](https://python.langchain.com/api_reference/experimental/text_splitter/langchain_experimental.text_splitter.SemanticChunker.html#semanticchunker)\n", + "\n", + "Split the text based on semantic similarity." + ], + "metadata": { + "id": "a8Rt52AepNNk" + } + }, + { + "cell_type": "code", + "source": [ + "# Semantic Chunking\n", + "\n", + "!pip install --quiet langchain_experimental langchain_openai\n", + "\n", + "import os\n", + "from langchain_experimental.text_splitter import SemanticChunker\n", + "from langchain_openai.embeddings import OpenAIEmbeddings\n", + "\n", + "# Add OpenAI API key as environment variable\n", + "os.environ[\"OPENAI_API_KEY\"] = \"sk-****\"\n", + "\n", + "with open(\"state_of_the_union.txt\") as f:\n", + " state_of_the_union = f.read()\n", + "\n", + "text_splitter = SemanticChunker(OpenAIEmbeddings())\n", + "\n", + "docs = text_splitter.create_documents([state_of_the_union])\n", + "print(docs[0].page_content)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "oHDYFAHeOjPA", + "outputId": "f7a5bbc2-c432-4370-bbf5-e529f4ff8c77" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\n", + "\n", + "Our Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty.\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Splitting by Tokens\n", + "\n", + "[Langchain tiktoken encoder](https://python.langchain.com/api_reference/text_splitters/character/langchain_text_splitters.character.CharacterTextSplitter.html#langchain_text_splitters.character.CharacterTextSplitter.from_tiktoken_encoder)\n", + "\n", + "Text splitter that uses tiktoken encoder to count length." + ], + "metadata": { + "id": "dV7RMi7_pRWn" + } + }, + { + "cell_type": "code", + "source": [ + "# Splits by Tokens\n", + "\n", + "# Using Tiktoken\n", + "!pip install --upgrade --quiet tiktoken\n", + "\n", + "with open(\"state_of_the_union.txt\") as f:\n", + " state_of_the_union = f.read()\n", + "\n", + "from langchain_text_splitters import CharacterTextSplitter\n", + "\n", + "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n", + " chunk_size=100, chunk_overlap=0\n", + ")\n", + "texts = text_splitter.split_text(state_of_the_union)\n", + "\n", + "print(texts[0])" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "7_WVw_kEQmJg", + "outputId": "93f501ed-d8a4-4350-a670-a6af25d2879d" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "WARNING:langchain_text_splitters.base:Created a chunk of size 123, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 104, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 109, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 106, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 129, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 111, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 118, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 132, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 231, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 177, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 112, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 130, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 116, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 184, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 139, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 112, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 151, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 203, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 138, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 123, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 213, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 134, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 130, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 125, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 139, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 111, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 130, which is longer than the specified 100\n", + "WARNING:langchain_text_splitters.base:Created a chunk of size 124, which is longer than the specified 100\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:\n", + "\n", + "Our Constitution declares that from time to time, the president shall give to Congress information about the state of our union. For 220 years, our leaders have fulfilled this duty. They have done so during periods of prosperity and tranquility. And they have done so in the midst of war and depression; at moments of great strife and great struggle.\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [], + "metadata": { + "id": "vMYrBTIvvGEg" + }, + "execution_count": null, + "outputs": [] } - ] - }, - { - "cell_type": "code", - "source": [], - "metadata": { - "id": "vMYrBTIvvGEg" - }, - "execution_count": null, - "outputs": [] - } - ] + ] } \ No newline at end of file diff --git a/tutorials/cli-sdk-to-convert-image-datasets-to-lance/main.ipynb b/tutorials/cli-sdk-to-convert-image-datasets-to-lance/main.ipynb index a5be18c9..4f2c2fc6 100644 --- a/tutorials/cli-sdk-to-convert-image-datasets-to-lance/main.ipynb +++ b/tutorials/cli-sdk-to-convert-image-datasets-to-lance/main.ipynb @@ -1,246 +1,259 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "UByKm8Q6dCEB" - }, - "source": [ - "### Imports" - ] + "cells": [ + { + "cell_type": "markdown", + "source": [ + "## Convert any Image dataset to Lance\n", + "\n", + "This notebook demonstrates for transforming any Image Dataset into the Lance format. It provides a straightforward solution for converting diverse image datasets into a standardized Lance format.\n", + "\n", + "" + ], + "metadata": { + "id": "KkwnfLYfEOHZ" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UByKm8Q6dCEB" + }, + "source": [ + "### Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "J8jICxVldCEC" + }, + "outputs": [], + "source": [ + "import os\n", + "import pandas as pd\n", + "import pyarrow as pa\n", + "import lance\n", + "import time\n", + "from tqdm import tqdm\n", + "\n", + "import warnings\n", + "\n", + "warnings.simplefilter(\"ignore\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AfUoIr4TdCEF" + }, + "source": [ + "### Set the variable according to your Image dataset\n", + "\n", + "Assign the path to your image dataset to the variable `image_dataset`. This dataset should contain your images organized into training, testing, and validation folders. These images will be used to convert them into Lance format.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5JvY3ucxdCEF" + }, + "outputs": [], + "source": [ + "image_dataset = \"image_dataset\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4ULduqF2dCEG" + }, + "source": [ + "### Processing the Images\n", + "\n", + "The `process_images` function is the central component of this notebook, responsible for transforming images from the training, testing, and validation folders into Lance format. This format typically includes essential attributes such as `image`, `filename`, `category`, and `data_type`.\n", + "\n", + "Specifically, `image` represents the actual image data, `filename` denotes the name of the file, `category` indicates the category to which the image belongs, and `data_type` specifies whether the image is from the training, testing, or validation set." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "S1wX3JVmdCEG" + }, + "outputs": [], + "source": [ + "def process_images():\n", + " # Get the current directory path\n", + " current_dir = os.getcwd()\n", + " images_folder = os.path.join(current_dir, image_dataset)\n", + " print(images_folder)\n", + "\n", + " # Define schema for RecordBatch\n", + " schema = pa.schema(\n", + " [\n", + " (\"image\", pa.binary()),\n", + " (\"filename\", pa.string()),\n", + " (\"category\", pa.string()),\n", + " (\"data_type\", pa.string()),\n", + " ]\n", + " )\n", + "\n", + " # Iterate over the data types (train, test, valid)\n", + " for data_type in [\"train\", \"test\", \"val\"]:\n", + " data_type_folder = os.path.join(images_folder, data_type)\n", + "\n", + " # Iterate over the categories within each data type\n", + " for category in os.listdir(data_type_folder):\n", + " category_folder = os.path.join(data_type_folder, category)\n", + "\n", + " # Iterate over the images within each category\n", + " for filename in tqdm(\n", + " os.listdir(category_folder), desc=f\"Processing {data_type} - {category}\"\n", + " ):\n", + " # Construct the full path to the image\n", + " image_path = os.path.join(category_folder, filename)\n", + "\n", + " # Read and convert the image to a binary format\n", + " with open(image_path, \"rb\") as f:\n", + " binary_data = f.read()\n", + "\n", + " image_array = pa.array([binary_data], type=pa.binary())\n", + " filename_array = pa.array([filename], type=pa.string())\n", + " category_array = pa.array([category], type=pa.string())\n", + " data_type_array = pa.array([data_type], type=pa.string())\n", + "\n", + " # Yield RecordBatch for each image\n", + " yield pa.RecordBatch.from_arrays(\n", + " [image_array, filename_array, category_array, data_type_array],\n", + " schema=schema,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Gcc4JvAYdCEI" + }, + "source": [ + "### Creating a Lance Dataset\n", + "\n", + "This function, `write_to_lance`, is designed to convert a PyArrow Table into a Lance dataset. It begins by defining the schema for the Lance dataset, specifying fields such as `image`, `filename`, `category`, and `data_type` , make sure the schema is the same as the one defined in the `process_images` function.\n", + "\n", + "Once the schema is established, the function determines the path for saving the Lance file, leveraging the current working directory and the provided `image_dataset` variable. It then initializes a RecordBatchReader using the defined schema and the data obtained from the `process_images` function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "IhfZI177dCEJ" + }, + "outputs": [], + "source": [ + "# Function to write PyArrow Table to Lance dataset\n", + "def write_to_lance():\n", + " # Create an empty RecordBatchIterator\n", + " schema = pa.schema(\n", + " [\n", + " pa.field(\"image\", pa.binary()),\n", + " pa.field(\"filename\", pa.string()),\n", + " pa.field(\"category\", pa.string()),\n", + " pa.field(\"data_type\", pa.string()),\n", + " ]\n", + " )\n", + "\n", + " # Specify the path where you want to save the Lance file\n", + " current_dir = os.getcwd()\n", + " images_folder = os.path.join(current_dir, image_dataset)\n", + " lance_file_path = os.path.join(images_folder, f\"{image_dataset}.lance\")\n", + "\n", + " reader = pa.RecordBatchReader.from_batches(schema, process_images())\n", + " lance.write_dataset(\n", + " reader,\n", + " lance_file_path,\n", + " schema,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tW9GeYJddCEL" + }, + "source": [ + "### Load a Lance Dataset and Visualize it in Pandas Dataframe" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UQHGcmzPdCEO" + }, + "source": [ + "`loading_into_pandas` function is designed to load a Lance dataset into a Pandas dataframe. It let's you see your Lance dataset in a pandas dataframe.\n", + "\n", + "The function takes the path to the Lance file as an argument and returns a pandas dataframe. Make sure the schema is the same as the one defined during the Lance dataset generation, refer to `process_images` function and also make sure the path to the Lance file is correct." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4wwa4FQmdCEP" + }, + "outputs": [], + "source": [ + "def loading_into_pandas():\n", + " # Load Lance file from the same folder\n", + " current_dir = os.getcwd()\n", + " images_folder = os.path.join(current_dir, image_dataset)\n", + " uri = os.path.join(images_folder, \"image_dataset.lance\")\n", + "\n", + " ds = lance.dataset(uri)\n", + "\n", + " # Accumulate data from batches into a list\n", + " data = []\n", + " for batch in tqdm(\n", + " ds.to_batches(\n", + " columns=[\"image\", \"filename\", \"category\", \"data_type\"], batch_size=10\n", + " ),\n", + " desc=\"Loading batches\",\n", + " ):\n", + " tbl = batch.to_pandas()\n", + " data.append(tbl)\n", + "\n", + " # Concatenate all DataFrames into a single DataFrame\n", + " df = pd.concat(data, ignore_index=True)\n", + " print(\"Pandas DataFrame is ready\")\n", + " print(\"Total Rows: \", df.shape[0])" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "env", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.2" + }, + "colab": { + "provenance": [] + } }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "J8jICxVldCEC" - }, - "outputs": [], - "source": [ - "import os\n", - "import pandas as pd\n", - "import pyarrow as pa\n", - "import lance\n", - "import time\n", - "from tqdm import tqdm\n", - "\n", - "import warnings\n", - "\n", - "warnings.simplefilter(\"ignore\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "AfUoIr4TdCEF" - }, - "source": [ - "### Set the variable according to your Image dataset\n", - "\n", - "Assign the path to your image dataset to the variable `image_dataset`. This dataset should contain your images organized into training, testing, and validation folders. These images will be used to convert them into Lance format.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5JvY3ucxdCEF" - }, - "outputs": [], - "source": [ - "image_dataset = \"image_dataset\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4ULduqF2dCEG" - }, - "source": [ - "### Processing the Images\n", - "\n", - "The `process_images` function is the central component of this notebook, responsible for transforming images from the training, testing, and validation folders into Lance format. This format typically includes essential attributes such as `image`, `filename`, `category`, and `data_type`.\n", - "\n", - "Specifically, `image` represents the actual image data, `filename` denotes the name of the file, `category` indicates the category to which the image belongs, and `data_type` specifies whether the image is from the training, testing, or validation set." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "S1wX3JVmdCEG" - }, - "outputs": [], - "source": [ - "def process_images():\n", - " # Get the current directory path\n", - " current_dir = os.getcwd()\n", - " images_folder = os.path.join(current_dir, image_dataset)\n", - " print(images_folder)\n", - "\n", - " # Define schema for RecordBatch\n", - " schema = pa.schema(\n", - " [\n", - " (\"image\", pa.binary()),\n", - " (\"filename\", pa.string()),\n", - " (\"category\", pa.string()),\n", - " (\"data_type\", pa.string()),\n", - " ]\n", - " )\n", - "\n", - " # Iterate over the data types (train, test, valid)\n", - " for data_type in [\"train\", \"test\", \"val\"]:\n", - " data_type_folder = os.path.join(images_folder, data_type)\n", - "\n", - " # Iterate over the categories within each data type\n", - " for category in os.listdir(data_type_folder):\n", - " category_folder = os.path.join(data_type_folder, category)\n", - "\n", - " # Iterate over the images within each category\n", - " for filename in tqdm(\n", - " os.listdir(category_folder), desc=f\"Processing {data_type} - {category}\"\n", - " ):\n", - " # Construct the full path to the image\n", - " image_path = os.path.join(category_folder, filename)\n", - "\n", - " # Read and convert the image to a binary format\n", - " with open(image_path, \"rb\") as f:\n", - " binary_data = f.read()\n", - "\n", - " image_array = pa.array([binary_data], type=pa.binary())\n", - " filename_array = pa.array([filename], type=pa.string())\n", - " category_array = pa.array([category], type=pa.string())\n", - " data_type_array = pa.array([data_type], type=pa.string())\n", - "\n", - " # Yield RecordBatch for each image\n", - " yield pa.RecordBatch.from_arrays(\n", - " [image_array, filename_array, category_array, data_type_array],\n", - " schema=schema,\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Gcc4JvAYdCEI" - }, - "source": [ - "### Creating a Lance Dataset\n", - "\n", - "This function, `write_to_lance`, is designed to convert a PyArrow Table into a Lance dataset. It begins by defining the schema for the Lance dataset, specifying fields such as `image`, `filename`, `category`, and `data_type` , make sure the schema is the same as the one defined in the `process_images` function.\n", - "\n", - "Once the schema is established, the function determines the path for saving the Lance file, leveraging the current working directory and the provided `image_dataset` variable. It then initializes a RecordBatchReader using the defined schema and the data obtained from the `process_images` function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "IhfZI177dCEJ" - }, - "outputs": [], - "source": [ - "# Function to write PyArrow Table to Lance dataset\n", - "def write_to_lance():\n", - " # Create an empty RecordBatchIterator\n", - " schema = pa.schema(\n", - " [\n", - " pa.field(\"image\", pa.binary()),\n", - " pa.field(\"filename\", pa.string()),\n", - " pa.field(\"category\", pa.string()),\n", - " pa.field(\"data_type\", pa.string()),\n", - " ]\n", - " )\n", - "\n", - " # Specify the path where you want to save the Lance file\n", - " current_dir = os.getcwd()\n", - " images_folder = os.path.join(current_dir, image_dataset)\n", - " lance_file_path = os.path.join(images_folder, f\"{image_dataset}.lance\")\n", - "\n", - " reader = pa.RecordBatchReader.from_batches(schema, process_images())\n", - " lance.write_dataset(\n", - " reader,\n", - " lance_file_path,\n", - " schema,\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "tW9GeYJddCEL" - }, - "source": [ - "### Load a Lance Dataset and Visualize it in Pandas Dataframe" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "UQHGcmzPdCEO" - }, - "source": [ - "`loading_into_pandas` function is designed to load a Lance dataset into a Pandas dataframe. It let's you see your Lance dataset in a pandas dataframe.\n", - "\n", - "The function takes the path to the Lance file as an argument and returns a pandas dataframe. Make sure the schema is the same as the one defined during the Lance dataset generation, refer to `process_images` function and also make sure the path to the Lance file is correct." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "4wwa4FQmdCEP" - }, - "outputs": [], - "source": [ - "def loading_into_pandas():\n", - " # Load Lance file from the same folder\n", - " current_dir = os.getcwd()\n", - " images_folder = os.path.join(current_dir, image_dataset)\n", - " uri = os.path.join(images_folder, \"image_dataset.lance\")\n", - "\n", - " ds = lance.dataset(uri)\n", - "\n", - " # Accumulate data from batches into a list\n", - " data = []\n", - " for batch in tqdm(\n", - " ds.to_batches(\n", - " columns=[\"image\", \"filename\", \"category\", \"data_type\"], batch_size=10\n", - " ),\n", - " desc=\"Loading batches\",\n", - " ):\n", - " tbl = batch.to_pandas()\n", - " data.append(tbl)\n", - "\n", - " # Concatenate all DataFrames into a single DataFrame\n", - " df = pd.concat(data, ignore_index=True)\n", - " print(\"Pandas DataFrame is ready\")\n", - " print(\"Total Rows: \", df.shape[0])" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "env", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.2" - }, - "colab": { - "provenance": [] - } - }, - "nbformat": 4, - "nbformat_minor": 0 + "nbformat": 4, + "nbformat_minor": 0 } \ No newline at end of file diff --git a/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb b/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb index ba1a040b..d44c2743 100644 --- a/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb +++ b/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb @@ -1,13515 +1,13564 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "provenance": [], - "gpuType": "T4" - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "language_info": { - "name": "python" - }, - "accelerator": "GPU", - "widgets": { - "application/vnd.jupyter.widget-state+json": { - "db0da7cffbea4d97a21968560229cc63": { - "model_module": "@jupyter-widgets/controls", - "model_name": "VBoxModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "VBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "VBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_68be93836c054703b4da20fe1d24146b", - "IPY_MODEL_085a92d31c874f9a8bc120ee0f5553c8", - "IPY_MODEL_19c133334a2d4f8e863da22ba8e36727", - "IPY_MODEL_56021216344347f8ad0d91787bf42074" - ], - "layout": "IPY_MODEL_5dbc8bd6ddd5408499e3a69fd1e421f5" - } - }, - "127213df2edb4289b139e64ebfbb935f": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_82956148127e4286abbae222449f0ce4", - "placeholder": "", - "style": "IPY_MODEL_5fb09536c5774f649a898950c3729088", - "value": "
Copy a token from
Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file.
Copy a token from
Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file.
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
\n",
- " \n",
- " \n"
+ "metadata": {
+ "id": "E29dFIihKlxW",
+ "outputId": "de95a5e3-2513-44fb-af3b-a9b4278383fe",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ }
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "{'question': 'What is your return policy?',\n",
+ " 'answer': 'Our return policy allows you to return products within 30 days of '\n",
+ " 'purchase for a full refund, provided they are in their original '\n",
+ " 'condition and packaging. Please refer to our Returns page for '\n",
+ " 'detailed instructions.'}\n"
+ ]
+ }
]
- },
- "metadata": {},
- "execution_count": 11
- }
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "we are using a sharded model the advantage of using sharded model is when you combine that with accelerate like for example in this case the model has been sharded into multiple pieces like more than like 14 pieces approximately so that helps accelerate to take a particular piece and then move it to different parts of the memory sometimes GPU memory sometimes CPU memory so that helps you\n",
- "\n",
- "load a very large model and also fine tune a large model in a smaller amount of memory that you have caught that's why we are using the sharded model"
- ],
- "metadata": {
- "id": "ZXUqtrpcMpDN"
- }
- },
- {
- "cell_type": "markdown",
- "source": [
- "### Changing the quantization type\n",
- "\n",
- "The 4bit integration comes with 2 different quantization types: FP4 and NF4. The NF4 dtype stands for Normal Float 4 and is introduced in the [QLoRA paper](https://arxiv.org/abs/2305.14314)\n",
- "\n",
- "YOu can switch between these two dtype using `bnb_4bit_quant_type` from `BitsAndBytesConfig`. By default, the FP4 quantization is used."
- ],
- "metadata": {
- "id": "wEXXPFrOaSZz"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "# Load Llama-7b bf16 shared Model & Tokenizer\n",
- "\n",
- "MODEL_NAME = \"TinyPixel/Llama-2-7B-bf16-sharded\"\n",
- "\n",
- "bnb_config = BitsAndBytesConfig(\n",
- " load_in_4bit=True,\n",
- " bnb_4bit_quant_type=\"nf4\", #\n",
- " bnb_4bit_compute_dtype=torch.bfloat16,\n",
- ")\n",
- "\n",
- "# model\n",
- "model = AutoModelForCausalLM.from_pretrained(\n",
- " MODEL_NAME,\n",
- " trust_remote_code=True,\n",
- " quantization_config=bnb_config,\n",
- ")\n",
- "\n",
- "# tockenizer\n",
- "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n",
- "tokenizer.pad_token = tokenizer.eos_token"
- ],
- "metadata": {
- "id": "-v9TmkByVVSn",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 862,
- "referenced_widgets": [
- "7e376b45695948239817b8e5195b0265",
- "53f8306f762c40208bd7ecfac6ea9fe4",
- "6ae60739c3f440738905090310f8a861",
- "5a2d250e95d149c5b0b48a9002426321",
- "4727720977fc46269e4deada686c5c4f",
- "759cd1bc90ad4b368f075b242d1770d5",
- "5ba79457f39a4092a1a7da4819c544f0",
- "cb086d61be174c7391223cd9278500a2",
- "618e6c61e6cc44c2948ce80ddb9a50ba",
- "4fa2a6de891849319290c2402dd93f32",
- "46f5065a034041d4ba1236d405f13c66",
- "c81fea829eb84022a6e462d786dec34c",
- "1a8b3120850d4b3896f31e7fca6a736e",
- "1e9dcf25e49b4c97bfd3256328f4ac82",
- "d14e56c713a54fbd838670a2dccc3cb6",
- "b00804101e7443f2b477e9faaf606810",
- "921309e80c024a76a90a54a918c344f6",
- "d53d6040c3074d1881dd3c1792fc8656",
- "52c97d6930b54b4d87be048c4121cfdd",
- "000bdae16fec436d83edb0e0b19f7248",
- "acd9a195cd31483ba90252f12b9b30b3",
- "e9131760ade24fcf8fec1de32afda46c",
- "729e45577f1a46cebb0416ffc401fbd4",
- "7afbe388f27b42fd93aa0828008087eb",
- "164d66ad6c054ba7908ee86fe2f74a6e",
- "89d34c9f97f54d55ab63d664e69cbf83",
- "a3453ba657654ab1bdba56d6ddfbe796",
- "1100ac4fdd82422a855d1e0fc675fd41",
- "08a2161434af4a1eaa5b612bfb12413b",
- "742fd80539204da0a599e6844a440dd3",
- "b690c22dc34f4b74817a8c454a6f0a6c",
- "813f87700d924cc2b3ae88047f8e4f11",
- "3584515e4d254db1a3e737a187e70594",
- "8ddfd3bcde8f4993aeb279db4ba8c715",
- "8be6272f61b04f8d850a4c852af2987b",
- "d7667212b28640e998c8980d0925699f",
- "9fcd72c7dc394faa87f9d4afea9a935d",
- "5514c80df18340c6ac6f1f5760aef2b9",
- "736baab93b1445e794372cab9b6faf18",
- "b0e8c0b5f6f54dfe8ff2d730424dfd33",
- "1ae51a8caed749b8b2458cc67782a852",
- "95e95c4526cb4c3888d8f1ba09dbb4ae",
- "1340163f1a694c068f14ec8f8eaed76f",
- "adcaac2088ab4b618617c9e483e27395",
- "68c57995c72c437d93abec28e97f662e",
- "07c8aed30b2b423fb64021a8d1250520",
- "e35fe935e6e34442a727a8035e04a8e4",
- "79efbd99303b4c95b1479cfd91b2569d",
- "fe45154a516e4273bf4055466e10d78f",
- "29b450c77f1941eeb6435e5a44c78c2f",
- "4c113de390d14a57b429b87db2aa8dbe",
- "018b438798774bd7a1ae582c5abd3485",
- "cf81d87c3fb04b31948fbe00b08002d3",
- "ef92c17bb3b6461d85aed96cf89ab07f",
- "e36df3d115844b518fd6f58b16a7dfd5",
- "7933b6e8fd8846e58f5610af96a5bc02",
- "c66d66807c57475aa64ce4a476e79dfe",
- "2e9f22501ce64989954f99043a3bc4c7",
- "e7164efc23f14fbca4adf6015abf2c03",
- "70802a0837a54dfb8925ad1233955207",
- "35f9130b437d4742b232692cf9df6d24",
- "b248895d24c949fcb10751359e7dc093",
- "a7c62cab596a473a874127158b8c9429",
- "916524fe436a436681d2faa14cb6910c",
- "60f6f492435f4eae9fc3dda2abf0cb1e",
- "47c8baac239a44d99e321c666c09c662",
- "e30c583f86584d67b64b4b97f249f6de",
- "2ea63b0b946f42a2b05a826acf71a85a",
- "e01284eb49744e56811f524c4957b7ac",
- "5f75a0480d734099929d2b3a35429d04",
- "498d30011222411d85e7378fa386ce4d",
- "41f5b6b898204da78a80bd0a51962345",
- "5b4c3ae017a641158399df89b479488d",
- "deede946a64a42138a7192d3624a2454",
- "745a6199154e4eb596624809a80c9b44",
- "3745c1609b8742cab64c87dc0c1a7ae7",
- "275fdcea1d6047ff9b2783f940673fd2",
- "f12b38ff9a804cad83b969b967778f0a",
- "ebfb497e967a4164b71b76c47410c547",
- "def0dd87d80c46a1b8e6fedf074ae8e0",
- "82132875e3ad4a0a9569644b4c40ed93",
- "e382df177b9f403abba003a15930209d",
- "988c4541b01d4d949597b23899bc6bb4",
- "a672cc307f6343bb90f25b678d3b9e53",
- "3ad5f8e829e94dc8b7646317e1bd26e0",
- "c9498a51bd4c428ca97ec0fe652e2d55",
- "c4896f188e35473bbde42c0ef0d5311a",
- "5edf1ffd7a194333b329b15c00d72589",
- "75210ffe4d3846dfa71158c8af3f4bdc",
- "969e5875225f4f7e82711e9d1a508889",
- "3bf30262624644a38d8c14adc3b6341d",
- "c35250df27284962bf4ea9dd49766a7c",
- "d8b240ebf8954351963ae9c13eea1146",
- "f6a703af8d0b4e2db0a1f10316300f85",
- "e85916bdce554d2aa47d24865bef8e7c",
- "113d85b0351942b48122b649b0dbe1de",
- "bd3ee08051b449ac8a4b5b547bf4a76c",
- "d24dc54ed22548538ec1d9a2fb5cce53",
- "124f9dff9c804e9dbabb5a8fafd82603",
- "8b77bf1730dc4e0caa034ec75d9c8a93",
- "1926a0a06f4345479d3ed257414f7e03",
- "8a3494842bca4e528ed9c143ea41ce1a",
- "b40067357d4a4d6691a9a83b3e48490e",
- "b312f83f5f8c4d51b6f68de7516202fc",
- "ca0fa43f6cfb493f9ff341ac4d05c884",
- "7fb17605c91847f08a8deaef91b7f20c",
- "1c95c715444a46068edc7d05067d2c41",
- "a68ecb19feea494ba254c0eb7e66fe4f",
- "9a5dec01480740449bd17585068c013a",
- "ba02fea9e6f0495bba5b87c1a135cc26",
- "6c5af26338fb4079a82bb0201b68efde",
- "f8f722ac1a3442719f4cd96305b94d99",
- "89410337c62e4e29a6bc3b84c735375f",
- "f9f55d83fd4249fd9c8e200e1cd6aada",
- "7114cf4f77d04df9805c5207943dac32",
- "b122dc1e8a8649d1be3b7716611608a7",
- "1ee3253d4a714f9db8f9b8d5ff9d9d13",
- "a6a3c4e60e484d12b42eeb5893802202",
- "5df4025cbb714a1b9e9f2ceaa0ff5519",
- "441e65db5e3c43f7a0f9b51231114f7b",
- "135198ada2894674b651cf6ea47ebe65",
- "024c62e1a0e94c68b5c585a018adde2b",
- "a6750217ac9a4766b0aabcab77544872",
- "1ef4852c2e4d447e8b666e48937cd482",
- "1ccfce2abd1e402090f0b0e90bf76ff6",
- "aedb84fa458040adae8e6d3613ac8547",
- "bc84f464a79b45c783bd8afeee6fee8f",
- "38fe5bf0a60648628443bde1dacd3228",
- "a2f4ad8e9d0140fd9f97ea48c57eef87",
- "d3ff9a4c593b42bb940662646b0086cb",
- "8e20584f3b934e979607e5a2ee743fdf",
- "de6bbff52e7e4fe3ba8fba87a2145e19",
- "e77f3c96ca4a42a6969108c23d59b6d6",
- "107322a867614535a794746898c9f9f2",
- "582d7adb0bfe4abe851c85b8f41e39b4",
- "21546d5c83f7461284b58de1b06b70a5",
- "e4a0e97acdec42759f2db3ba2069bbde",
- "4d4e64061acf4f2787839e07d72ef4f5",
- "b00c114a99e74cc0aa02913be667ca88",
- "a1d25f07e0534dddb12cadf9529e9432",
- "36001d04af14412ea24cc0eaee568af3",
- "55f5bc5cf02e4818b099f9fcb310cf52",
- "2554aa220e7b45f58bab1973eeec052a",
- "0101774ee5b04733816b37bed0a8718f",
- "de9f5a374a7e4c4ea6004537b5fa592f",
- "d6c854d084994108a80e0c222b9cb40c",
- "02379b2c6671496ebb2708470219bfa4",
- "9c95304c7d6d4d33a8d67f8474c6ca1f",
- "934a0c90544f40d09da8587a15b169a3",
- "8f5e05b4428c487dbc1d85ee8392fdc0",
- "2f1a8d6f72364760a9d1cf9a3c2077cf",
- "010091aac936471baaf540866333e905",
- "fe4479df735e4ce7bd118a377e7d3d30",
- "a924a11b71714d1a987681c1493d3696",
- "d57d692cf9f1472d99d4e89d0e03d9eb",
- "bbdd1c96b9b5404cb79fe7fae4d128e3",
- "9670ee9091c14561ae4f90a644fded61",
- "528c1a53b5b6421ba7a5597047cb425f",
- "41c33ef46e144825b304c2cddd7a820a",
- "4843ed950e19430e832c4c3ffe171872",
- "4c3ba306bf49409a95e1f20697f4b939",
- "08e978370b4c46f5b3c9f6766d39c475",
- "6dfb7f632c6b46bb95ab3fab5a25731b",
- "9c60639463664238a15c45922810c1ef",
- "f70fa89c94d54856a977b0d115f1b16f",
- "2409a58a1a184e44997cd08309087bb7",
- "bcca7f5c95e64a299b2c566b49ea9d8e",
- "2664f5896c7548c98dec4adacafc1773",
- "1a0b776d0683427ab3c2d0336090fc07",
- "da42b875ac5c445b8568f08deb70f2d7",
- "8bd6df78e23946c2a654996a7bb151d1",
- "99d810d349ea45f3b5974d546451716d",
- "092068270e1645b58a8199df31830104",
- "0772fe28de6f4ede9be2e009168d6440",
- "4ed3dd6adde6471c9f3bb968a9042a6c",
- "445a34b3b3cc4d79a5439fc31327d401",
- "db9dbbf55ac6453190a8e9d3276415c7",
- "ac86550d992f4f52872d239def42f7f7",
- "1062d77445a645ba85733911ef5492ea",
- "e43a2e9cdc744425bf8da2e572075e6f",
- "5fc541b7b8684a80afa5b93bea656518",
- "78e2fafee89644ef88e3929c9dce257e",
- "0063a1c656d3466cac6f94b44b685150",
- "6500bbf2ed8348aa84ea6c8a6e2d5f56",
- "5f8d3a8fcb7b42998210cefe90762ac7",
- "65537c62866f4e8fa740b574a4e79e4b",
- "9caa5c6e2c1547029a8473e5c1e97873",
- "c650fd87ef804ee3b6775d2afbacc8a2",
- "93b4cfcf4c3347068d050ed0879163bb",
- "921c5285939944e6af199235bc84e283",
- "a907179506f3484da21438176d320441",
- "ab23442cd7484bc6a4873c0492bfc130",
- "8d49d6edb84d476382f397981eff1955",
- "fce620ead2124b3292f440a586d7c352",
- "6474ee1f52654f4e8f2483bb7e17cb4f",
- "8d480f4ddf0c4ad49e08dcc3d3978756",
- "5918b970798e4131bf871ffcdf3d1a3e",
- "e6a7791cdfb5438c9fb607f0b86d7a77",
- "d0e5256910d04cc8a5f6a27cb0ae9314",
- "b54f855f1df0422c874bda0da5be30a0",
- "e1d671173a094e6ca82d5d98c913a34a",
- "b680bbe6bb924703aa81966c44fb44c5",
- "100d854f151e4c1ab19a57442dc0c648",
- "0ef96e0ad97f4b2180e91e649aec4b14",
- "cef32900d66147bc83861e367de53d43",
- "b8a501f289524d1ab413e2dbd7054e8a",
- "57e878a0f0814255ab877a5b99f0ba46",
- "6651654128c8494487523de9e9462f93",
- "d4aa32a889a14029ad6824dce5b552ac",
- "e968170092424db885496476020c19a3",
- "6dd7063dda0b4ba2810f968b388b103b",
- "815e5297e36843268d52f87181627e54",
- "3894e6fca1d24eb7ad4215291bf17c69",
- "d297a2d53ae24ae6be67380086aecf67",
- "0f8a83e4c96e475ebf8fff5a0a74825a",
- "a2b35805c80b491298b2e3079b72f99b",
- "de03fb55b67c4040a06b38c63b482203",
- "ccbf46622cf14879bb6d72bf3cee40a8",
- "0cc2d95d223244df8f7a37cee028aa5c",
- "3aabb7373794469f831f692b70f0ce39",
- "34795cc3d44d4ed3b52839d815aaa71e",
- "352f9ecf4fc64a4fae9d9cc507848865",
- "d6260183890e4cbfb821387e76743c53",
- "1e04fae4e50f400bbc4a027e9623abce",
- "4f7c42616b2d4abd86af3f1feb81984e",
- "d0de28d203c34cfca54e95a2d39994f7",
- "614805637cbc43c2bebfa56196e3b7b1",
- "4b4566c507ce4f9292f8d121776ccd55",
- "9c6c6fc03fc04b94b6300aaed02c336a",
- "9328eb252a5442cbbc61076da40d43fe",
- "53c519a74a5a4ebc89b4e5d8a4fc355c",
- "556b1e5bb1364189acc6ab399320e869",
- "fd86c77c01464ffcb897bcd2747c649e",
- "4d44fdce5825457ba23ffbe0d6461a9e",
- "a01a51db34a0413f9a02ff390f706858",
- "ec59b989de8543f3b31c04fb2ade85f8",
- "14f7fa4cf8494aacbf914eae68033b88",
- "3b8fe7d02a8f41c2ae2df89c98b36845",
- "420c0ca380194243baf2d35c32a74cf0",
- "0fb08803a958442ea9b3e811f3d613fd",
- "ac215a2f754a42f0bb184deac83b5591",
- "4faf605688c646f2bd002fd086b60aa2",
- "90a73d86ca8e4c77aea19bccb8fd654b",
- "b32a5eae40274ed3ba224a5244ee901b",
- "502c7b36cf6649b1a7e4c38a394e6712",
- "71c3b8925ee54952b93e02a736f8bb59",
- "2ad724c423ba4c7bb0fffb4c1ff6bbc7",
- "e4d09be6e10244679d93f324a475db7c",
- "47714bd097074f53b3840afc7cd84074",
- "2f7ffa207bda4402b46859fe41a14dfa",
- "dc4873a72b04410ea60264066129b821",
- "3a7da27e98124c4e908bcca0e1fe1e21",
- "0630de2afbf54b96a0144a66dfae67d0"
- ]
},
- "outputId": "1242b674-f165-4e2a-aa62-6e1b51918c40"
- },
- "execution_count": 12,
- "outputs": [
{
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
- "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
- "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
- "You will be able to reuse this secret in all of your notebooks.\n",
- "Please note that authentication is recommended but still optional to access public models or datasets.\n",
- " warnings.warn(\n"
- ]
+ "cell_type": "code",
+ "source": [
+ "# reading dataset.json for Q&A\n",
+ "with open(\"dataset.json\", \"w\") as f:\n",
+ " json.dump(data[\"questions\"], f)"
+ ],
+ "metadata": {
+ "id": "9cE25RS5Mk1D"
+ },
+ "execution_count": null,
+ "outputs": []
},
{
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "config.json: 0%| | 0.00/626 [00:00\n",
+ " | \n", - " | question | \n", - "answer | \n", - "
|---|---|---|
| 0 | \n", - "How can I create an account? | \n", - "To create an account, click on the 'Sign Up' b... | \n", - "
| 1 | \n", - "What payment methods do you accept? | \n", - "We accept major credit cards, debit cards, and... | \n", - "
| 2 | \n", - "How can I track my order? | \n", - "You can track your order by logging into your ... | \n", - "
| 3 | \n", - "What is your return policy? | \n", - "Our return policy allows you to return product... | \n", - "
| 4 | \n", - "Can I cancel my order? | \n", - "You can cancel your order if it has not been s... | \n", - "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
\n",
+ " \n",
+ " \n"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 11
+ }
+ ]
},
{
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "model.safetensors.index.json: 0%| | 0.00/28.1k [00:00"
+ ],
+ "text/html": [
+ "\n",
+ " | \n", + " | question | \n", + "answer | \n", + "
|---|---|---|
| 0 | \n", + "How can I create an account? | \n", + "To create an account, click on the 'Sign Up' b... | \n", + "
| 1 | \n", + "What payment methods do you accept? | \n", + "We accept major credit cards, debit cards, and... | \n", + "
| 2 | \n", + "How can I track my order? | \n", + "You can track your order by logging into your ... | \n", + "
| 3 | \n", + "What is your return policy? | \n", + "Our return policy allows you to return product... | \n", + "
| 4 | \n", + "Can I cancel my order? | \n", + "You can cancel your order if it has not been s... | \n", + "
\n",
+ " \n",
+ " \n",
+ " [80/80 07:21, Epoch 4/5]\n",
+ "
\n",
+ " | Step | \n", + "Training Loss | \n", + "
|---|---|
| 1 | \n", + "2.275600 | \n", + "
| 2 | \n", + "2.245900 | \n", + "
| 3 | \n", + "1.933500 | \n", + "
| 4 | \n", + "1.858800 | \n", + "
| 5 | \n", + "2.012600 | \n", + "
| 6 | \n", + "1.801800 | \n", + "
| 7 | \n", + "1.794000 | \n", + "
| 8 | \n", + "1.489300 | \n", + "
| 9 | \n", + "1.587700 | \n", + "
| 10 | \n", + "1.560400 | \n", + "
| 11 | \n", + "1.471600 | \n", + "
| 12 | \n", + "1.551800 | \n", + "
| 13 | \n", + "1.598900 | \n", + "
| 14 | \n", + "1.403500 | \n", + "
| 15 | \n", + "1.195500 | \n", + "
| 16 | \n", + "1.334300 | \n", + "
| 17 | \n", + "1.191300 | \n", + "
| 18 | \n", + "1.072000 | \n", + "
| 19 | \n", + "1.151500 | \n", + "
| 20 | \n", + "1.109000 | \n", + "
| 21 | \n", + "1.135800 | \n", + "
| 22 | \n", + "1.122000 | \n", + "
| 23 | \n", + "0.953200 | \n", + "
| 24 | \n", + "1.027600 | \n", + "
| 25 | \n", + "0.940800 | \n", + "
| 26 | \n", + "0.907100 | \n", + "
| 27 | \n", + "0.784400 | \n", + "
| 28 | \n", + "0.880200 | \n", + "
| 29 | \n", + "1.014100 | \n", + "
| 30 | \n", + "0.843800 | \n", + "
| 31 | \n", + "1.039000 | \n", + "
| 32 | \n", + "0.733400 | \n", + "
| 33 | \n", + "0.676000 | \n", + "
| 34 | \n", + "0.628600 | \n", + "
| 35 | \n", + "0.906400 | \n", + "
| 36 | \n", + "0.530600 | \n", + "
| 37 | \n", + "0.678700 | \n", + "
| 38 | \n", + "0.595400 | \n", + "
| 39 | \n", + "0.748500 | \n", + "
| 40 | \n", + "0.590200 | \n", + "
| 41 | \n", + "0.563200 | \n", + "
| 42 | \n", + "0.639400 | \n", + "
| 43 | \n", + "0.513500 | \n", + "
| 44 | \n", + "0.645800 | \n", + "
| 45 | \n", + "0.542300 | \n", + "
| 46 | \n", + "0.364400 | \n", + "
| 47 | \n", + "0.481800 | \n", + "
| 48 | \n", + "0.647700 | \n", + "
| 49 | \n", + "0.489400 | \n", + "
| 50 | \n", + "0.634600 | \n", + "
| 51 | \n", + "0.365600 | \n", + "
| 52 | \n", + "0.420700 | \n", + "
| 53 | \n", + "0.487100 | \n", + "
| 54 | \n", + "0.533600 | \n", + "
| 55 | \n", + "0.361700 | \n", + "
| 56 | \n", + "0.460900 | \n", + "
| 57 | \n", + "0.515300 | \n", + "
| 58 | \n", + "0.547600 | \n", + "
| 59 | \n", + "0.514300 | \n", + "
| 60 | \n", + "0.547600 | \n", + "
| 61 | \n", + "0.409700 | \n", + "
| 62 | \n", + "0.347000 | \n", + "
| 63 | \n", + "0.467800 | \n", + "
| 64 | \n", + "0.429700 | \n", + "
| 65 | \n", + "0.441100 | \n", + "
| 66 | \n", + "0.406900 | \n", + "
| 67 | \n", + "0.505200 | \n", + "
| 68 | \n", + "0.405800 | \n", + "
| 69 | \n", + "0.427400 | \n", + "
| 70 | \n", + "0.528000 | \n", + "
| 71 | \n", + "0.290200 | \n", + "
| 72 | \n", + "0.301500 | \n", + "
| 73 | \n", + "0.484300 | \n", + "
| 74 | \n", + "0.383900 | \n", + "
| 75 | \n", + "0.444400 | \n", + "
| 76 | \n", + "0.424000 | \n", + "
| 77 | \n", + "0.486000 | \n", + "
| 78 | \n", + "0.480600 | \n", + "
| 79 | \n", + "0.397400 | \n", + "
| 80 | \n", + "0.419400 | \n", + "
" + ] + }, + "metadata": {} + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "TrainOutput(global_step=80, training_loss=0.8391439635306597, metrics={'train_runtime': 447.6633, 'train_samples_per_second': 0.715, 'train_steps_per_second': 0.179, 'total_flos': 649997819142144.0, 'train_loss': 0.8391439635306597, 'epoch': 4.05})" + ] + }, + "metadata": {}, + "execution_count": 27 + } + ] }, { - "output_type": "display_data", - "data": { - "text/plain": [ - "tokenizer.json: 0%| | 0.00/1.84M [00:00" + "cell_type": "code", + "source": [ + "%%time\n", + "prompt = f\"\"\"\n", + ": How can I create an account?\n", + ":\n", + "\"\"\".strip()\n", + "\n", + "encoding = tokenizer(prompt, return_tensors=\"pt\").to(DEVICE)\n", + "with torch.inference_mode():\n", + " outputs = model.generate(\n", + " input_ids=encoding.input_ids,\n", + " attention_mask=encoding.attention_mask,\n", + " generation_config=generation_config,\n", + "\n", + " )\n", + "print(tokenizer.decode(outputs[0], skip_special_tokens=True))" ], - "text/html": [ - "\n", - "
\n",
- " \n",
- " \n",
- " [80/80 07:21, Epoch 4/5]\n",
- "
\n",
- " | Step | \n", - "Training Loss | \n", - "
|---|---|
| 1 | \n", - "2.275600 | \n", - "
| 2 | \n", - "2.245900 | \n", - "
| 3 | \n", - "1.933500 | \n", - "
| 4 | \n", - "1.858800 | \n", - "
| 5 | \n", - "2.012600 | \n", - "
| 6 | \n", - "1.801800 | \n", - "
| 7 | \n", - "1.794000 | \n", - "
| 8 | \n", - "1.489300 | \n", - "
| 9 | \n", - "1.587700 | \n", - "
| 10 | \n", - "1.560400 | \n", - "
| 11 | \n", - "1.471600 | \n", - "
| 12 | \n", - "1.551800 | \n", - "
| 13 | \n", - "1.598900 | \n", - "
| 14 | \n", - "1.403500 | \n", - "
| 15 | \n", - "1.195500 | \n", - "
| 16 | \n", - "1.334300 | \n", - "
| 17 | \n", - "1.191300 | \n", - "
| 18 | \n", - "1.072000 | \n", - "
| 19 | \n", - "1.151500 | \n", - "
| 20 | \n", - "1.109000 | \n", - "
| 21 | \n", - "1.135800 | \n", - "
| 22 | \n", - "1.122000 | \n", - "
| 23 | \n", - "0.953200 | \n", - "
| 24 | \n", - "1.027600 | \n", - "
| 25 | \n", - "0.940800 | \n", - "
| 26 | \n", - "0.907100 | \n", - "
| 27 | \n", - "0.784400 | \n", - "
| 28 | \n", - "0.880200 | \n", - "
| 29 | \n", - "1.014100 | \n", - "
| 30 | \n", - "0.843800 | \n", - "
| 31 | \n", - "1.039000 | \n", - "
| 32 | \n", - "0.733400 | \n", - "
| 33 | \n", - "0.676000 | \n", - "
| 34 | \n", - "0.628600 | \n", - "
| 35 | \n", - "0.906400 | \n", - "
| 36 | \n", - "0.530600 | \n", - "
| 37 | \n", - "0.678700 | \n", - "
| 38 | \n", - "0.595400 | \n", - "
| 39 | \n", - "0.748500 | \n", - "
| 40 | \n", - "0.590200 | \n", - "
| 41 | \n", - "0.563200 | \n", - "
| 42 | \n", - "0.639400 | \n", - "
| 43 | \n", - "0.513500 | \n", - "
| 44 | \n", - "0.645800 | \n", - "
| 45 | \n", - "0.542300 | \n", - "
| 46 | \n", - "0.364400 | \n", - "
| 47 | \n", - "0.481800 | \n", - "
| 48 | \n", - "0.647700 | \n", - "
| 49 | \n", - "0.489400 | \n", - "
| 50 | \n", - "0.634600 | \n", - "
| 51 | \n", - "0.365600 | \n", - "
| 52 | \n", - "0.420700 | \n", - "
| 53 | \n", - "0.487100 | \n", - "
| 54 | \n", - "0.533600 | \n", - "
| 55 | \n", - "0.361700 | \n", - "
| 56 | \n", - "0.460900 | \n", - "
| 57 | \n", - "0.515300 | \n", - "
| 58 | \n", - "0.547600 | \n", - "
| 59 | \n", - "0.514300 | \n", - "
| 60 | \n", - "0.547600 | \n", - "
| 61 | \n", - "0.409700 | \n", - "
| 62 | \n", - "0.347000 | \n", - "
| 63 | \n", - "0.467800 | \n", - "
| 64 | \n", - "0.429700 | \n", - "
| 65 | \n", - "0.441100 | \n", - "
| 66 | \n", - "0.406900 | \n", - "
| 67 | \n", - "0.505200 | \n", - "
| 68 | \n", - "0.405800 | \n", - "
| 69 | \n", - "0.427400 | \n", - "
| 70 | \n", - "0.528000 | \n", - "
| 71 | \n", - "0.290200 | \n", - "
| 72 | \n", - "0.301500 | \n", - "
| 73 | \n", - "0.484300 | \n", - "
| 74 | \n", - "0.383900 | \n", - "
| 75 | \n", - "0.444400 | \n", - "
| 76 | \n", - "0.424000 | \n", - "
| 77 | \n", - "0.486000 | \n", - "
| 78 | \n", - "0.480600 | \n", - "
| 79 | \n", - "0.397400 | \n", - "
| 80 | \n", - "0.419400 | \n", - "
" + "metadata": { + "id": "TScya6W0RaDL", + "outputId": "f4e599b4-9674-4f92-a2e7-7aabc66f28a3", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + ": How can I create an account?\n", + ": To create an account, click on the 'Sign Up' button on the top right corner of the website. Follow the instructions to complete the registration process.\n", + ": You can place an order by adding items to your shopping cart and proceeding to\n", + "CPU times: user 4.37 s, sys: 252 ms, total: 4.62 s\n", + "Wall time: 4.68 s\n" + ] + } ] - }, - "metadata": {} }, { - "output_type": "execute_result", - "data": { - "text/plain": [ - "TrainOutput(global_step=80, training_loss=0.8391439635306597, metrics={'train_runtime': 447.6633, 'train_samples_per_second': 0.715, 'train_steps_per_second': 0.179, 'total_flos': 649997819142144.0, 'train_loss': 0.8391439635306597, 'epoch': 4.05})" - ] - }, - "metadata": {}, - "execution_count": 27 - } - ] - }, - { - "cell_type": "markdown", - "source": [ - "I trained it for 100 epochs, and as you can observe, the loss consistently decreases, indicating room for further improvement. Consider extending the training to a higher number of epochs for potential enhancements" - ], - "metadata": { - "id": "NKLu3hYCYWbW" - } - }, - { - "cell_type": "markdown", - "source": [ - "## save the model" - ], - "metadata": { - "id": "Bm-4ny2SYgYz" - } - }, - { - "cell_type": "code", - "source": [ - "model.save_pretrained(\"trained-model\")" - ], - "metadata": { - "id": "YQ4VipiaQ38Q" - }, - "execution_count": 28, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "## Push the model in Hugging face" - ], - "metadata": { - "id": "gEWFr9vKYbvI" - } - }, - { - "cell_type": "code", - "source": [ - "model.push_to_hub(\"Prasant/Llama2-7b-qlora-chat-support-bot-faq\")" - ], - "metadata": { - "id": "bvkTqEZFQ3_W", - "outputId": "4d5e4b9e-a8ec-4514-e8e6-a390dfa39dd9", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 85, - "referenced_widgets": [ - "c91ff953b85847fd9bcbc952b259d13e", - "80055b5100d0410ab59ba6c000e5f65c", - "eb05122c60014feca974b9be59a89570", - "d3a6a5bcce6a4e7eaa2db04a53f5b933", - "c5ccd8c9f94749d391fb0815180cc7ed", - "64d26285c5d5413dab6ff9649e5b3b11", - "100084059b6d41858478218c3ffec02a", - "157518129a4d46c2b23b86223ae34ee4", - "e44f488773994690a9505db3c7f5ad6f", - "d6e500ae93e04beb8d5b80162df8cf9a", - "4f4ac78e20d64c12a4d3d394f7ab78ae" - ] - } - }, - "execution_count": 29, - "outputs": [ + "cell_type": "code", + "source": [ + "# helper function to generate responses\n", + "def generate_response(question: str) -> str:\n", + " prompt = f\"\"\"\n", + ": {question}\n", + ":\n", + "\"\"\".strip()\n", + " encoding = tokenizer(prompt, return_tensors=\"pt\").to(DEVICE)\n", + " with torch.inference_mode():\n", + " outputs = model.generate(\n", + " input_ids=encoding.input_ids,\n", + " attention_mask=encoding.attention_mask,\n", + " generation_config=generation_config,\n", + " )\n", + " response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n", + "\n", + " assistant_start = \":\"\n", + " response_start = response.find(assistant_start)\n", + " return response[response_start + len(assistant_start) :].strip()" + ], + "metadata": { + "id": "6ZQc3XvhRaGa" + }, + "execution_count": null, + "outputs": [] + }, { - "output_type": "display_data", - "data": { - "text/plain": [ - "adapter_model.safetensors: 0%| | 0.00/134M [00:00 str:\n", - " prompt = f\"\"\"\n", - ": {question}\n", - ":\n", - "\"\"\".strip()\n", - " encoding = tokenizer(prompt, return_tensors=\"pt\").to(DEVICE)\n", - " with torch.inference_mode():\n", - " outputs = model.generate(\n", - " input_ids=encoding.input_ids,\n", - " attention_mask=encoding.attention_mask,\n", - " generation_config=generation_config,\n", - " )\n", - " response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n", - "\n", - " assistant_start = \":\"\n", - " response_start = response.find(assistant_start)\n", - " return response[response_start + len(assistant_start) :].strip()" - ], - "metadata": { - "id": "6ZQc3XvhRaGa" - }, - "execution_count": 38, - "outputs": [] - }, - { - "cell_type": "code", - "source": [ - "# prompt\n", - "prompt = \"Question: Can I return a product if it was a clearance or final sale item?\"\n", - "print(generate_response(prompt))" - ], - "metadata": { - "id": "IXiY44KGRaJv", - "outputId": "357e5eaa-f72c-4025-d8a7-6e739e70ad8a", - "colab": { - "base_uri": "https://localhost:8080/" - } - }, - "execution_count": 42, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Question: Can I return a product if it was a clearance or final sale item?\n", - ": Clearance or final sale items are typically non-returnable. Please refer to the product description or contact our customer support team for specific return instructions.\n", - ": You can request a return by contacting our customer support team. We will provide you with\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "# prompt\n", - "prompt = \"Question: What happens when I return a clearance item?\"\n", - "print(generate_response(prompt))" - ], - "metadata": { - "id": "yJE1BrB8RaM8", - "outputId": "83ae00c0-bdff-40f0-ccfb-d37d6348cf61", - "colab": { - "base_uri": "https://localhost:8080/" - } - }, - "execution_count": 43, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Question: What happens when I return a clearance item?\n", - ": Clearance items are non-refundable and non-exchangeable. However, you can request a store credit for the full value of the item. Please contact our customer support team for assistance.\n", - ": We accept returns within 30 days\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "# prompt\n", - "prompt = \"Question: How do I know when I'll receive my order?\"\n", - "print(generate_response(prompt))" - ], - "metadata": { - "id": "sDkq0PbhRw-2", - "outputId": "be1753a4-2e91-42fb-dac6-cae6830594d4", - "colab": { - "base_uri": "https://localhost:8080/" } - }, - "execution_count": 44, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Question: How do I know when I'll receive my order?\n", - ": Once you place an order, we will send you a confirmation email with your order details and estimated delivery time. You can track your order's progress by logging into your account or checking your order confirmation email.\n", - ": If you need to\n" - ] - } - ] - }, - { - "cell_type": "code", - "source": [ - "################ falcon with lama2\n", - "# https://github.com/curiousily/Get-Things-Done-with-Prompt-Engineering-and-LangChain/blob/master/07.falcon-qlora-fine-tuning.ipynb" - ], - "metadata": { - "id": "3L3UxKlVRaPy" - }, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": [ - "At this time of writing, the models that support accelerate are:\n", - "\n", - "\n", - "[\n", - " 'bigbird_pegasus', 'blip_2', 'bloom', 'bridgetower', 'codegen', 'deit', 'esm',\n", - " 'gpt2', 'gpt_bigcode', 'gpt_neo', 'gpt_neox', 'gpt_neox_japanese', 'gptj', 'gptsan_japanese',\n", - " 'lilt', 'llama', 'longformer', 'longt5', 'luke', 'm2m_100', 'mbart', 'mega', 'mt5', 'nllb_moe',\n", - " 'open_llama', 'opt', 'owlvit', 'plbart', 'roberta', 'roberta_prelayernorm', 'rwkv', 'switch_transformers',\n", - " 't5', 'vilt', 'vit', 'vit_hybrid', 'whisper', 'xglm', 'xlm_roberta'\n", - "]\n" - ], - "metadata": { - "id": "fspW3KiWaaCO" - } - } - ] + ] } \ No newline at end of file