cleanup (#880)

* first

* fix pinecone package name

* adjust progress and add dummy provider delay option

* cleanup phase 1

* adding more sections to api docs

* added more pages

* Add backwards compatible Record.feedback_results .

* version bump

* misssed version bump

* bump py_script quickstarts

* typo

* finished first pass over Tru class doc standardization.

* a bit more

* nits

* another nit

* more

* fix links

* nonetype for python 3.9

* fix class nav not working

* update generate_score and generate_score_and_reason docstrings

* remove bad js

* adding record to api docs

* adjust providers nav and pages

* forgot one provider

* move things

* piepline fixes

---------

Co-authored-by: Josh Reini <[email protected]>
Co-authored-by: Josh Reini <[email protected]>

.gitignore

@@ -43,5 +43,6 @@ credentials.toml
 
 **/*.sqlite
 
-# Virtual environment
-.venv
+# Virtual environments
+.venv
+.conda

.style.yapf

@@ -2,3 +2,5 @@
 based_on_style = google
 DEDENT_CLOSING_BRACKETS=true
 SPLIT_BEFORE_FIRST_ARGUMENT=true
+SPLIT_COMPLEX_COMPREHENSION=true
+COLUMN_LIMIT=80

Makefile

@@ -1,9 +1,31 @@
 SHELL := /bin/bash
-CONDA_ENV := demo3
-CONDA := source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate $(CONDA_ENV)
+CONDA := source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate
 
-format:
-	$(CONDA); bash format.sh
+# Create the conda env for building website, docs, formatting, etc.
+.conda/docs:
+	conda create python=3.11 --yes --prefix=.conda/docs
+	$(CONDA) .conda/docs; \
+	pip install -r trulens_eval/trulens_eval/requirements.txt; \
+	pip install -r trulens_eval/trulens_eval/requirements.optional.txt; \
+	pip install -r docs/docs_requirements.txt
 
+# Run the code formatter.
+format: .conda/docs
+	$(CONDA) .conda/docs; bash format.sh
+
+# Start a jupyer lab instance.
 lab:
-	$(CONDA); jupyter lab --ip=0.0.0.0 --no-browser --ServerApp.token=deadbeef
+	$(CONDA) .conda/docs; jupyter lab --ip=0.0.0.0 --no-browser --ServerApp.token=deadbeef
+
+# Serve the documentation website.
+serve: site
+	$(CONDA) .conda/docs; mkdocs serve -a 127.0.0.1:8000
+
+# Build the documentation website.
+site: .conda/docs $(shell find docs -type f) mkdocs.yml
+	$(CONDA) .conda/docs; mkdocs build
+	rm -Rf site/overrides
+
+# Check that links in the documentation are valid. Requires the lychee tool.
+linkcheck: site
+	lychee "site/**/*.html"

README.md

@@ -9,7 +9,7 @@ README.md . If you are editing README.md, your changes will be overwritten.
 ![GitHub](https://img.shields.io/github/license/truera/trulens)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/trulens_eval)
 [![Slack](https://img.shields.io/badge/slack-join-green?logo=slack)](https://communityinviter.com/apps/aiqualityforum/josh)
-[![Docs](https://img.shields.io/badge/docs-trulens.org-blue)](https://www.trulens.org/welcome/)
+[![Docs](https://img.shields.io/badge/docs-trulens.org-blue)](https://www.trulens.org/install/)
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/releases/rc-trulens-eval-0.17.0/trulens_eval/examples/quickstart/colab/langchain_quickstart_colab.ipynb)
 
 # 🦑 **Welcome to TruLens!**

docs/docs_requirements.txt

@@ -1,9 +1,13 @@
-mkdocs==1.4.2
-python-markdown-math==0.8
-mkdocs-material==9.1.4
-mkdocstrings-python
-mkdocstrings==0.20.0
-mkdocs-jupyter
-jupyter_contrib_nbextensions
-notebook<7
-markdown-include
+# Packages required for building documentation pages.
+
+mkdocs               >= 1.5.3
+mkdocs-material      >= 9.5.8
+mkdocstrings-python  >= 1.8.0
+mkdocstrings         >= 0.24.0
+mkdocs-jupyter       >= 0.24.6
+
+python-markdown-math >= 0.8
+markdown-include     >= 0.8.1
+
+jupyter_contrib_nbextensions >= 0.7.0
+notebook <7

docs/nn-explainability.html

@@ -15,30 +15,27 @@
 
     <meta property="og:title" content="TruLens">
     <meta property="og:description" content="Explainability for Neural Networks">
-    <meta property="og:image" content="https://www.trulens.org/assets/image/Neural_Network_Explainability.png">
+    <meta property="og:image" content="https://www.trulens.org/assets/images/Neural_Network_Explainability.png">
     <meta property="og:url" content="https://www.trulens.org">
 
-    <link rel="apple-touch-icon" sizes="180x180" href="Assets/favicon/apple-touch-icon.png">
-    <link rel="icon" type="image/png" sizes="32x32" href="Assets/favicon/favicon-32x32.png">
-    <link rel="icon" type="image/png" sizes="16x16" href="Assets/favicon/favicon-16x16.png">
+    <link rel="apple-touch-icon" sizes="180x180" href="assets/favicon/apple-touch-icon.png">
+    <link rel="icon" type="image/png" sizes="32x32" href="assets/favicon/favicon-32x32.png">
+    <link rel="icon" type="image/png" sizes="16x16" href="assets/favicon/favicon-16x16.png">
     <link rel="manifest" href="/Assets/favicon/site.webmanifest">
     <link rel="mask-icon" href="/Assets/favicon/safari-pinned-tab.svg" color="#034a61">
     <meta name="msapplication-TileColor" content="#034a61">
     <meta name="theme-color" content="#034a61">
 
     <link rel="stylesheet" href="/stylesheets/style.css">
 
-    <link rel="stylesheet" href="../stylesheets/style.css">
-
     <link rel="preconnect" href="https://fonts.googleapis.com">
-    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
     <link
         href="https://fonts.googleapis.com/css2?family=Source+Sans+Pro:wght@300;400;700&display=swap&family=Space+Grotesk:wght@400;500;700&display=swap"
         rel="stylesheet">
 
     <script src="/Assets/Scripts/app.js"></script>
 
-    <link rel="stylesheet" href="../Assets/Scripts/app.js">
+    <link rel="stylesheet" href="/Assets/Scripts/app.js">
 
     <!-- Google Tag Manager -->
     <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':

docs/overrides/home.html

@@ -15,26 +15,25 @@
 
     <meta property="og:title" content="TruLens">
     <meta property="og:description" content="Explainability for Neural Networks">
-    <meta property="og:image" content="https://www.trulens.org/assets/image/Neural_Network_Explainability.png">
+    <meta property="og:image" content="/assets/images/Neural_Network_Explainability.png">
     <meta property="og:url" content="https://www.trulens.org">
 
-    <link rel="apple-touch-icon" sizes="180x180" href="Assets/favicon/apple-touch-icon.png">
-    <link rel="icon" type="image/png" sizes="32x32" href="Assets/favicon/favicon-32x32.png">
-    <link rel="icon" type="image/png" sizes="16x16" href="Assets/favicon/favicon-16x16.png">
+    <link rel="apple-touch-icon" sizes="180x180" href="/assets/favicon/apple-touch-icon.png">
+    <link rel="icon" type="image/png" sizes="32x32" href="/assets/favicon/favicon-32x32.png">
+    <link rel="icon" type="image/png" sizes="16x16" href="/assets/favicon/favicon-16x16.png">
     <link rel="manifest" href="/Assets/favicon/site.webmanifest">
     <link rel="mask-icon" href="/Assets/favicon/safari-pinned-tab.svg" color="#034a61">
     <meta name="msapplication-TileColor" content="#034a61">
     <meta name="theme-color" content="#034a61">
 
-    <link rel="stylesheet" href="../stylesheets/style.css">
+    <link rel="stylesheet" href="stylesheets/style.css">
 
     <link rel="preconnect" href="https://fonts.googleapis.com">
-    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
     <link
         href="https://fonts.googleapis.com/css2?family=Source+Sans+Pro:wght@300;400;700&display=swap&family=Space+Grotesk:wght@400;500;700&display=swap"
         rel="stylesheet">
 
-    <link rel="stylesheet" href="../Assets/Scripts/app.js">
+    <link rel="stylesheet" href="/Assets/Scripts/app.js">
 
     <!-- Google Tag Manager -->
     <script>(function (w, d, s, l, i) {
@@ -117,7 +116,7 @@
                         <a href="https://go.truera.com/newsletter-archive" target="blank">
                             <li>Newsletter</li>
                         </a>
-                        <a href="https://www.trulens.org/trulens_eval/install/" target="_blank"
+                        <a href="trulens_eval/install/" target="_blank"
                             class="header__btn nav__btn d-if fd-r ai-c jc-c mt-xxl-mob">
                             <svg class="hide-desktop" width="24" height="25" viewBox="0 0 24 25" fill="none"
                                 xmlns="http://www.w3.org/2000/svg">
@@ -283,8 +282,8 @@ <h3>Test</h3>
 
                 <div class="ta-c">
                     <h2 class="ta-l">How it works</h2>
-                    <img class="hide-mobile" src="../img/trulens-diagram.svg" alt="TruLens diagram" />
-                    <img class="hide-desktop" src="../img/trulens-diagram-mob.png" alt="TruLens diagram vertical" />
+                    <img class="hide-mobile" src="img/trulens-diagram.svg" alt="TruLens diagram" />
+                    <img class="hide-desktop" src="img/trulens-diagram-mob.png" alt="TruLens diagram vertical" />
                 </div>
 
             </section>
@@ -384,7 +383,7 @@ <h3>Drive rapid iteration with scalable, programmatic feedback.</h3>
                 </div>
                 <div class="grid-columns-2 box-dark">
                     <div class="box-violet">
-                        <img class="" src="../img/trulens-feedback-functions.png" alt="TruLens Feednacl Functions" />
+                        <img class="" src="img/trulens-feedback-functions.png" alt="TruLens Feednacl Functions" />
                     </div>
                     <div class="box teal">
                         <h3>Get the breadth of feedback you need to evaluate app performance.</h3>
@@ -548,7 +547,7 @@ <h2>Get started using TruLens today</h2>
                             </svg>
                         </a>
                         <p class="strong font-primary">Documentation</p>
-                        <span>Read about the library <a href="https://www.trulens.org/trulens_eval/install/"
+                        <span>Read about the library <a href="trulens_eval/install/"
                                 target="_blank">here</a>.</span>
                     </div>
                     <div class="box__shadow">
@@ -629,7 +628,7 @@ <h3>What’s a Feedback Function?</h3>
                         </p>
                     </div>
                     <div class="full-image">
-                        <img class="" src="../img/trulens-more-functions.png"
+                        <img class="" src="img/trulens-more-functions.png"
                             alt="TruLens Unlimited Feedback Functions" />
                     </div>
                 </div>

docs/trulens_eval/STANDARDS.md

@@ -0,0 +1 @@
+../../trulens_eval/STANDARDS.md

docs/trulens_eval/api/app/app.md

@@ -0,0 +1,9 @@
+Apps in trulens derive from two classes,
+[AppDefinition][trulens_eval.schema.AppDefinition] and
+[App][trulens_eval.app.App]. The first contains only serialized or serializable
+components in a JSON-like format while the latter contains the executable apps
+that may or may not be serializable.
+
+::: trulens_eval.schema.AppDefinition
+
+::: trulens_eval.app.App

docs/trulens_eval/api/app/trubasicapp.md

@@ -0,0 +1,5 @@
+# Tru Basic App
+
+::: trulens_eval.tru_basic_app.TruBasicApp
+    options:
+      inherited_members: true

docs/trulens_eval/api/app/truchain.md

@@ -0,0 +1,5 @@
+# 🦜️🔗 Tru Chain
+
+::: trulens_eval.tru_chain.TruChain
+    options:
+      inherited_members: true

docs/trulens_eval/api/app/trucustom.md

@@ -0,0 +1,5 @@
+# Tru Custom App
+
+::: trulens_eval.tru_custom_app.TruCustomApp
+    options:
+      inherited_members: true

docs/trulens_eval/api/app/trullama.md

@@ -0,0 +1,5 @@
+# 🦙 Tru Llama
+
+::: trulens_eval.tru_llama.TruLlama
+    options:
+      inherited_members: true

docs/trulens_eval/api/app/truvirtual.md

@@ -0,0 +1,9 @@
+# Tru Virtual
+
+::: trulens_eval.tru_virtual.VirtualRecord
+
+::: trulens_eval.tru_virtual.VirtualApp
+
+::: trulens_eval.tru_virtual.TruVirtual
+    options:
+      inherited_members: true

docs/trulens_eval/api/db.md

@@ -0,0 +1,5 @@
+# Database
+
+::: trulens_eval.db
+
+::: trulens_eval.database.sqlalchemy_db

docs/trulens_eval/api/endpoint.md

@@ -0,0 +1,3 @@
+# Endpoint API
+
+::: trulens_eval.feedback.provider.endpoint.base.Endpoint

docs/trulens_eval/api/feedback.md

@@ -1,10 +1,9 @@
-# Feedback Function APIs
-Below are out of the box feedback functions and how to instantiate them.
+# Feedback
 
-::: trulens_eval.trulens_eval.feedback.provider.base
+::: trulens_eval.feedback.feedback.ImpCallable
 
-::: trulens_eval.trulens_eval.feedback.groundedness
+::: trulens_eval.feedback.feedback.AggCallable
 
-::: trulens_eval.trulens_eval.feedback.groundtruth
+::: trulens_eval.feedback.feedback.Feedback
 
-::: trulens_eval.trulens_eval.feedback.embeddings
+::: trulens_eval.schema.FeedbackDefinition

docs/trulens_eval/api/provider/azureopenai.md

@@ -0,0 +1,10 @@
+# AzureOpenAI Provider
+
+Below is how you can instantiate AzureOpenAI as a provider.
+
+Additionally, all feedback functions listed in these two classes can be run with AzureOpenAI:
+
+* [OpenAI class][trulens_eval.feedback.provider.openai.OpenAI]
+* [LLMProvider class][trulens_eval.feedback.provider.base.LLMProvider]
+
+::: trulens_eval.feedback.provider.openai.AzureOpenAI

docs/trulens_eval/api/provider/bedrock.md

@@ -0,0 +1,12 @@
+# AWS Bedrock Provider
+
+Below is how you can instantiate AWS Bedrock as a provider. [Amazon
+Bedrock](https://aws.amazon.com/bedrock/) is a fully managed service that makes
+FMs from leading AI startups and Amazon available via an API, so you can choose
+from a wide range of FMs to find the model that is best suited for your use case
+
+All feedback functions listed in the base [LLMProvider
+class][trulens_eval.feedback.provider.base.LLMProvider] can be run with AWS
+Bedrock.
+
+::: trulens_eval.feedback.provider.bedrock.Bedrock

docs/trulens_eval/api/provider/huggingface.md

@@ -0,0 +1,3 @@
+# 🤗 Huggingface Provider
+
+::: trulens_eval.feedback.provider.hugs.Huggingface