The LLM app requires both long-term and short-term memories. Long-term memory includes factual knowledge, historical facts, background stories etc. They are best added to the context as complete chapters instead of small chunks of text to maintain the internal consistency of the knowledge.
RAG
is an important technique to inject contextual knowledge into an LLM application. It improves accuracy and reduces the hallucination of LLMs.
@@ -46,7 +47,7 @@
For example, if you ask ChatGPT the question What is Layer 2, the answer is that Layer 2 is a concept from the computer network. However, if you ask a blockchain person, they answer that Layer 2 is a way to scale the original Ethereum network. That's the difference between a generic LLM and knowledge-supplemented LLMs.
-
We will cover the external knowledge preparation and how a knowledge-supplemented LLM completes a conversation. If you have learned how a RAG application works, go to Build a RAG application with Gaia to start building one.
+
We will cover the external knowledge preparation and how a knowledge-supplemented LLM completes a conversation. If you have learned how a RAG application works, go to Build a RAG application with Gaia to start building one.
Create embeddings for your own knowledge as the long-term memory.
Lifecycle of a user query on a knowledge-supplemented LLM.
In this section, we will discuss how to create a vector collection snapshot for optimal retrieval of
long-form text documents. The approach is to create two columns of text in a CSV file.
diff --git a/build/1.0.0/creator-guide/knowledge/firecrawl/index.html b/build/1.0.0/creator-guide/knowledge/firecrawl/index.html
index cdc9c633..88c82401 100644
--- a/build/1.0.0/creator-guide/knowledge/firecrawl/index.html
+++ b/build/1.0.0/creator-guide/knowledge/firecrawl/index.html
@@ -2,7 +2,7 @@
-
+
Knowledge base from a URL | Gaia
@@ -11,18 +11,19 @@
+
+
+
-
-
-
-
-
+
+
+
-
In this section, we will discuss how to create a vector collection snapshot from a Web URL. First, we will parse the URL to a structured markdown file. Then, we will follow the steps from Knowledge base from a markdown file to create embedding for your URL.
In this section, we will discuss how to create a vector collection snapshot from a markdown file. The
snapshot file can then be loaded by a Gaia node as its knowledge base.
The markdown file is segmented into multiple sections by headings. See an example. Each section is turned into a vector, and when
diff --git a/build/1.0.0/creator-guide/knowledge/pdf/index.html b/build/1.0.0/creator-guide/knowledge/pdf/index.html
index 500eca92..42aff690 100644
--- a/build/1.0.0/creator-guide/knowledge/pdf/index.html
+++ b/build/1.0.0/creator-guide/knowledge/pdf/index.html
@@ -2,7 +2,7 @@
-
+
Knowledge base from a PDF file | Gaia
@@ -11,18 +11,19 @@
+
+
+
-
-
-
-
-
+
+
+
-
In this section, we will discuss how to create a vector collection snapshot from a PDF file. First, we will parse the unstructured PDF file to a structured markdown file. Then, we will follow the steps from Knowledge base from a markdown file to create embedding for your PDF files.
In this section, we will discuss how to create a vector collection snapshot from a plain text file. The
snapshot file can then be loaded by a Gaia node as its knowledge base.
The text file is segmented into multiple chunks by blank lines. See an example. Each chunk is turned into a vector, and when
diff --git a/build/1.0.0/creator-guide/knowledge/web-tool/index.html b/build/1.0.0/creator-guide/knowledge/web-tool/index.html
index 1ab4a371..d4566455 100644
--- a/build/1.0.0/creator-guide/knowledge/web-tool/index.html
+++ b/build/1.0.0/creator-guide/knowledge/web-tool/index.html
@@ -2,7 +2,7 @@
-
+
Build a knowledge base using Gaia web tool | Gaia
@@ -11,18 +11,19 @@
+
+
+
-
-
-
-
-
+
+
+
-
This guide provides all the information you need to quickly set up and run a Gaia Domain.
Note: Ensure that you are the owner of a Gaia Domain Name before proceeding. You can verify your Gaia Domain Name in the "Assets" section of your profile.
GaiaNet is a decentralized computing infrastructure that enables everyone to create, deploy, scale, and monetize their own AI agents that reflect their styles, values, knowledge, and expertise.
It allows individuals and businesses to create AI agents. Each GaiaNet node provides:
-
-
-
+
+
+
-