Add automatic llms.txt generation via mkdocs-llmstxt plugin. Closes #8562

Author: aberoham

docs/README.md

@@ -72,4 +72,14 @@ This guide is for contributors looking to make changes to the documentation in t
 
 2. **Push your new changes on a new branch**: Feel free to add or edit existing documentation and open a PR for your changes. Once your PR is reviewed and approved, the changes will be ready to merge into main. 
 
-3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
+3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins.
+
+## LLMs.txt Files
+
+DSPy documentation automatically generates [llmstxt.org](https://llmstxt.org/) compliant files to help LLMs understand and work with DSPy:
+
+- **`/llms.txt`** - A structured overview with links to key documentation sections
+- **`/llms-full.txt`** - A comprehensive version with full content for deep understanding
+
+These txt files are automatically generated during the docs build process using the [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt) plugin. When modifying documentation, consider updating the `llmstxt` plugin section within `mkdocs.yml` to help LLMs better understand DSPy's capabilities.
+

docs/mkdocs.yml

@@ -236,6 +236,82 @@ plugins:
             "docs/quick-start/getting-started-02.md": "tutorials/rag/index.ipynb"
             "quick-start/getting-started-01.md": "tutorials/rag/index.ipynb"
             "quick-start/getting-started-02.md": "tutorials/rag/index.ipynb"
+    - llmstxt:
+        markdown_description: >
+          DSPy is the framework for programming—rather than prompting—language models.
+          DSPy unifies techniques for prompting, fine-tuning, reasoning, tool use, and evaluation of LMs.
+          It provides a systematic approach to building AI applications through composable modules, 
+          optimization techniques, and evaluation frameworks.
+        full_output: llms-full.txt
+        sections:
+          Getting Started:
+            - index.md: DSPy overview and quick start
+            - faqs.md: Frequently asked questions
+            - cheatsheet.md: DSPy cheatsheet
+          Core Learning:
+            - learn/index.md: Learn DSPy fundamentals
+            - learn/programming/*.md: Programming concepts - signatures, modules, language models
+            - learn/evaluation/*.md: Evaluation framework - data handling, metrics, overview
+            - learn/optimization/*.md: Optimization techniques and optimizers
+          Building AI Programs:
+            - tutorials/build_ai_program/index.md: Building AI programs overview
+            - tutorials/conversation_history/index.md: Managing conversation history
+            - tutorials/custom_module/index.ipynb: Customizing DSPy modules
+            - tutorials/classification/index.md: Classification with DSPy
+            - tutorials/rag/index.ipynb: Retrieval-Augmented Generation
+            - tutorials/agents/index.ipynb: Building AI agents
+            - tutorials/customer_service_agent/index.ipynb: Customer service agents
+            - tutorials/entity_extraction/index.ipynb: Entity extraction
+            - tutorials/multihop_search/index.ipynb: Multi-hop search and reasoning
+            - tutorials/program_of_thought/index.ipynb: Program of thought reasoning
+            - tutorials/tool_use/index.ipynb: Advanced tool use
+          Optimization and Training:
+            - tutorials/optimize_ai_program/index.md: Optimization overview
+            - tutorials/math/index.ipynb: Math reasoning optimization
+            - tutorials/classification_finetuning/index.ipynb: Classification fine-tuning
+            - tutorials/games/index.ipynb: Agent fine-tuning with games
+            - tutorials/rl_ai_program/index.md: Reinforcement learning overview
+            - tutorials/rl_papillon/index.ipynb: RL for privacy-conscious delegation
+            - tutorials/rl_multihop/index.ipynb: RL for multi-hop research
+          Specialized Use Cases:
+            - tutorials/papillon/index.md: Privacy-conscious delegation
+            - tutorials/image_generation_prompting/index.ipynb: Image generation prompting
+            - tutorials/audio/index.ipynb: Audio processing with DSPy
+            - tutorials/ai_text_game/index.md: Creative text-based AI games
+            - tutorials/email_extraction/index.md: Email information extraction
+            - tutorials/sample_code_generation/index.md: Code generation for libraries
+            - tutorials/llms_txt_generation/index.md: Generating llms.txt files
+            - tutorials/mem0_react_agent/index.md: Memory-enabled ReAct agents
+            - tutorials/yahoo_finance_react/index.md: Financial analysis agents
+          Development and Deployment:
+            - tutorials/core_development/index.md: Development workflow
+            - tutorials/mcp/index.md: Model Context Protocol integration
+            - tutorials/output_refinement/best-of-n-and-refine.md: Output refinement techniques
+            - tutorials/saving/index.md: Saving and loading models
+            - tutorials/cache/index.md: Caching strategies
+            - tutorials/deployment/index.md: Production deployment
+            - tutorials/observability/index.md: Debugging and observability
+            - tutorials/optimizer_tracking/index.md: Tracking optimizer performance
+            - tutorials/streaming/index.md: Streaming responses
+            - tutorials/async/index.md: Asynchronous processing
+            - production/index.md: DSPy in production environments
+          API Reference - Core:
+            - api/index.md: API documentation overview
+            - api/modules/*.md: Core modules - Predict, ChainOfThought, ReAct, etc.
+            - api/signatures/*.md: Signature system - InputField, OutputField, Signature
+            - api/primitives/*.md: Core primitives - Example, Prediction, History, etc.
+          API Reference - Optimizers:
+            - api/optimizers/*.md: All optimizers - BootstrapFewShot, COPRO, MIPROv2, etc.
+          API Reference - Adapters and Tools:
+            - api/adapters/*.md: Input/output adapters - ChatAdapter, JSONAdapter, etc.
+            - api/evaluation/*.md: Evaluation metrics and tools
+            - api/models/*.md: Language model interfaces
+            - api/tools/*.md: Built-in tools - ColBERTv2, Embeddings, PythonInterpreter
+            - api/utils/*.md: Utility functions and helpers
+          Community:
+            - community/community-resources.md: Community resources
+            - community/use-cases.md: Real-world use cases
+            - community/how-to-contribute.md: Contributing guidelines
 
 extra:
     social:

docs/requirements.txt

@@ -5,5 +5,6 @@ mkdocs-material[imaging]
 mkdocs-redirects
 mkdocstrings
 mkdocstrings-python
+mkdocs-llmstxt>=0.3.0
 urllib3==1.26.6
 mistune==3.0.2