diff --git a/docs/mcp/api/tools.md b/docs/mcp/api/tools.md new file mode 100644 index 0000000..ba95d22 --- /dev/null +++ b/docs/mcp/api/tools.md @@ -0,0 +1,687 @@ +# MCP Tools API Reference + +Complete reference documentation for all 43 tools available in the ContextFrame MCP server. + +## Tool Categories + +### [Document Operations](#document-operations) +Core CRUD operations for managing documents +- [`document_create`](#document_create) - Create new document +- [`document_get`](#document_get) - Retrieve document by ID +- [`document_update`](#document_update) - Update existing document +- [`document_delete`](#document_delete) - Delete document +- [`document_exists`](#document_exists) - Check if document exists +- [`document_list`](#document_list) - List all documents +- [`document_create_batch`](#document_create_batch) - Create multiple documents +- [`document_update_batch`](#document_update_batch) - Update multiple documents +- [`document_delete_batch`](#document_delete_batch) - Delete multiple documents + +### [Search Operations](#search-operations) +Advanced search capabilities +- [`search_documents`](#search_documents) - Text, vector, or hybrid search +- [`search_similar`](#search_similar) - Find similar documents +- [`search_by_metadata`](#search_by_metadata) - Filter by metadata +- [`search_within_collection`](#search_within_collection) - Collection-scoped search +- [`search_with_filters`](#search_with_filters) - Advanced filtering +- [`search_stream`](#search_stream) - Streaming search results + +### [Collection Management](#collection-management) +Organize documents into collections +- [`collection_create`](#collection_create) - Create new collection +- [`collection_get`](#collection_get) - Get collection details +- [`collection_update`](#collection_update) - Update collection +- [`collection_delete`](#collection_delete) - Delete collection +- [`collection_list`](#collection_list) - List all collections +- [`collection_add_documents`](#collection_add_documents) - Add documents to collection +- [`collection_remove_documents`](#collection_remove_documents) - Remove documents from collection +- [`collection_list_documents`](#collection_list_documents) - List documents in collection +- [`collection_stats`](#collection_stats) - Get collection statistics + +### [Analytics Tools](#analytics-tools) +Usage statistics and insights +- [`dataset_stats`](#dataset_stats) - Overall dataset statistics +- [`search_analytics`](#search_analytics) - Search usage patterns +- [`usage_metrics`](#usage_metrics) - API usage metrics +- [`cost_estimate`](#cost_estimate) - Estimate operation costs +- [`performance_metrics`](#performance_metrics) - Performance statistics + +### [Import/Export Tools](#importexport-tools) +Bulk data operations +- [`import_documents`](#import_documents) - Bulk import from file +- [`export_documents`](#export_documents) - Bulk export to file +- [`import_from_url`](#import_from_url) - Import from remote source +- [`export_to_cloud`](#export_to_cloud) - Export to cloud storage +- [`import_status`](#import_status) - Check import progress +- [`export_status`](#export_status) - Check export progress + +### [System Tools](#system-tools) +Health and management +- [`health_check`](#health_check) - Server health status +- [`list_tools`](#list_tools) - List available tools +- [`get_tool_info`](#get_tool_info) - Get tool details +- [`validate_dataset`](#validate_dataset) - Validate dataset integrity +- [`optimize_dataset`](#optimize_dataset) - Optimize performance +- [`clear_cache`](#clear_cache) - Clear server cache +- [`server_info`](#server_info) - Server configuration info + +### [Advanced Tools](#advanced-tools) +Specialized operations +- [`create_embedding`](#create_embedding) - Generate embeddings +- [`reindex_documents`](#reindex_documents) - Rebuild search indexes +- [`backup_dataset`](#backup_dataset) - Create dataset backup +- [`restore_dataset`](#restore_dataset) - Restore from backup + +--- + +## Document Operations + +### document_create + +Create a new document in the dataset. + +**Parameters:** +```json +{ + "id": "string (optional)", // Custom document ID + "content": "string (required)", // Document content + "metadata": "object (optional)", // Custom metadata + "embeddings": "array (optional)", // Pre-computed embeddings + "collection_id": "string (optional)", // Add to collection + "upsert": "boolean (optional)" // Update if exists (default: false) +} +``` + +**Returns:** +```json +{ + "id": "doc_abc123", + "created": true, + "message": "Document created successfully" +} +``` + +**Example:** +```bash +curl -X POST http://localhost:8000/mcp/v1/tools/document_create \ + -H "Content-Type: application/json" \ + -d '{ + "params": { + "content": "# Introduction to Machine Learning\n\nMachine learning is...", + "metadata": { + "title": "ML Introduction", + "author": "John Doe", + "tags": ["ml", "tutorial"] + } + } + }' +``` + +### document_get + +Retrieve a document by its ID. + +**Parameters:** +```json +{ + "id": "string (required)", // Document ID + "include_embeddings": "boolean", // Include embeddings (default: false) + "include_content": "boolean" // Include content (default: true) +} +``` + +**Returns:** +```json +{ + "id": "doc_abc123", + "content": "# Introduction to Machine Learning...", + "metadata": { + "title": "ML Introduction", + "author": "John Doe", + "created_at": "2024-01-15T10:00:00Z" + } +} +``` + +### document_update + +Update an existing document. + +**Parameters:** +```json +{ + "id": "string (required)", // Document ID + "content": "string (optional)", // New content + "metadata": "object (optional)", // New/updated metadata + "merge_metadata": "boolean", // Merge vs replace metadata (default: true) + "update_embeddings": "boolean" // Regenerate embeddings (default: true) +} +``` + +**Returns:** +```json +{ + "id": "doc_abc123", + "updated": true, + "message": "Document updated successfully" +} +``` + +### document_delete + +Delete a document from the dataset. + +**Parameters:** +```json +{ + "id": "string (required)", // Document ID + "permanent": "boolean" // Hard delete (default: false) +} +``` + +**Returns:** +```json +{ + "id": "doc_abc123", + "deleted": true, + "message": "Document deleted successfully" +} +``` + +### document_exists + +Check if a document exists. + +**Parameters:** +```json +{ + "id": "string (required)" // Document ID +} +``` + +**Returns:** +```json +{ + "exists": true, + "id": "doc_abc123" +} +``` + +### document_list + +List documents with optional filtering. + +**Parameters:** +```json +{ + "limit": "integer", // Max results (default: 100) + "offset": "integer", // Skip results (default: 0) + "collection_id": "string", // Filter by collection + "metadata_filter": "object", // Filter by metadata + "sort_by": "string", // Sort field (default: "created_at") + "sort_order": "string" // "asc" or "desc" (default: "desc") +} +``` + +**Returns:** +```json +{ + "documents": [ + { + "id": "doc_123", + "metadata": {...} + } + ], + "total_count": 1523, + "offset": 0, + "limit": 100 +} +``` + +### document_create_batch + +Create multiple documents in a single operation. + +**Parameters:** +```json +{ + "documents": [ // Array of documents + { + "content": "string", + "metadata": "object", + "id": "string (optional)" + } + ], + "collection_id": "string", // Add all to collection + "upsert": "boolean", // Update if exists + "batch_size": "integer" // Process in batches (default: 100) +} +``` + +**Returns:** +```json +{ + "created": 95, + "updated": 5, + "failed": 0, + "total": 100, + "document_ids": ["doc_1", "doc_2", ...] +} +``` + +--- + +## Search Operations + +### search_documents + +Search documents using text, vector, or hybrid search. + +**Parameters:** +```json +{ + "query": "string (required)", // Search query + "search_type": "string", // "text", "vector", "hybrid" (default: "hybrid") + "limit": "integer", // Max results (default: 10) + "offset": "integer", // Skip results (default: 0) + "collection_id": "string", // Search within collection + "metadata_filter": "object", // Filter by metadata + "min_score": "number", // Minimum relevance score + "rerank": "boolean" // Rerank results (default: false) +} +``` + +**Returns:** +```json +{ + "documents": [ + { + "id": "doc_123", + "content": "...", + "metadata": {...}, + "score": 0.95, + "highlights": ["matching text"] + } + ], + "total_count": 42, + "query": "machine learning", + "execution_time_ms": 23 +} +``` + +**Example:** +```python +# Text search +results = client.search_documents( + query="neural networks", + search_type="text", + limit=20 +) + +# Vector (semantic) search +results = client.search_documents( + query="How do transformers work?", + search_type="vector", + limit=10 +) + +# Hybrid search (combines text and vector) +results = client.search_documents( + query="BERT architecture", + search_type="hybrid", + metadata_filter={"type": "paper"} +) +``` + +### search_similar + +Find documents similar to a given document. + +**Parameters:** +```json +{ + "document_id": "string", // Reference document ID + "limit": "integer", // Max results (default: 10) + "min_similarity": "number", // Min similarity score (0-1) + "collection_id": "string", // Search within collection + "exclude_source": "boolean" // Exclude reference doc (default: true) +} +``` + +**Returns:** +```json +{ + "similar_documents": [ + { + "id": "doc_456", + "similarity": 0.92, + "metadata": {...} + } + ], + "source_document_id": "doc_123", + "total_found": 15 +} +``` + +### search_by_metadata + +Search using metadata filters with SQL-like syntax. + +**Parameters:** +```json +{ + "filters": "object (required)", // Filter conditions + "limit": "integer", // Max results + "offset": "integer", // Skip results + "sort_by": "string", // Sort field + "sort_order": "string" // "asc" or "desc" +} +``` + +**Filter Examples:** +```json +// Exact match +{"type": "paper"} + +// Multiple conditions (AND) +{"type": "paper", "year": 2024} + +// Operators +{"year": {"$gte": 2020}} +{"tags": {"$contains": "ml"}} +{"author": {"$in": ["Smith", "Jones"]}} +{"title": {"$regex": ".*learning.*"}} +``` + +--- + +## Collection Management + +### collection_create + +Create a new collection. + +**Parameters:** +```json +{ + "name": "string (required)", // Collection name + "description": "string", // Collection description + "metadata": "object", // Custom metadata + "id": "string" // Custom ID (optional) +} +``` + +**Returns:** +```json +{ + "id": "coll_abc123", + "name": "Research Papers", + "created": true +} +``` + +### collection_add_documents + +Add documents to a collection. + +**Parameters:** +```json +{ + "collection_id": "string (required)", // Collection ID + "document_ids": "array (required)", // Document IDs to add + "position": "string" // Position in collection +} +``` + +**Returns:** +```json +{ + "collection_id": "coll_abc123", + "added": 25, + "already_present": 5, + "not_found": 0 +} +``` + +### collection_stats + +Get detailed statistics for a collection. + +**Parameters:** +```json +{ + "collection_id": "string (required)" // Collection ID +} +``` + +**Returns:** +```json +{ + "id": "coll_abc123", + "name": "Research Papers", + "document_count": 1523, + "total_size_bytes": 15728640, + "created_at": "2024-01-01T00:00:00Z", + "last_updated": "2024-01-15T12:00:00Z", + "metadata": {...} +} +``` + +--- + +## Analytics Tools + +### dataset_stats + +Get comprehensive dataset statistics. + +**Parameters:** +```json +{ + "include_collections": "boolean", // Include collection stats + "include_storage": "boolean" // Include storage details +} +``` + +**Returns:** +```json +{ + "total_documents": 10523, + "total_collections": 42, + "storage_size_bytes": 1073741824, + "index_size_bytes": 268435456, + "unique_metadata_keys": ["type", "author", "year"], + "document_size_stats": { + "min": 100, + "max": 50000, + "avg": 2500, + "median": 2000 + } +} +``` + +### usage_metrics + +Get API usage metrics. + +**Parameters:** +```json +{ + "time_range": "string", // "hour", "day", "week", "month" + "group_by": "string" // "tool", "user", "hour" +} +``` + +**Returns:** +```json +{ + "time_range": "day", + "metrics": { + "total_requests": 15234, + "unique_users": 42, + "by_tool": { + "search_documents": 8523, + "document_create": 2341 + }, + "avg_response_time_ms": 45, + "error_rate": 0.001 + } +} +``` + +--- + +## Import/Export Tools + +### import_documents + +Import documents from various formats. + +**Parameters:** +```json +{ + "source": "string (required)", // File path or URL + "format": "string", // "json", "csv", "parquet" + "collection_id": "string", // Target collection + "mapping": "object", // Field mapping + "batch_size": "integer", // Import batch size + "on_error": "string" // "skip", "abort" (default: "skip") +} +``` + +**Returns:** +```json +{ + "import_id": "imp_123", + "status": "in_progress", + "processed": 500, + "total": 1000, + "errors": [] +} +``` + +### export_documents + +Export documents to various formats. + +**Parameters:** +```json +{ + "format": "string (required)", // "json", "csv", "parquet" + "destination": "string", // File path or URL + "document_ids": "array", // Specific documents + "collection_id": "string", // Export collection + "filters": "object", // Metadata filters + "include_embeddings": "boolean" // Include embeddings +} +``` + +**Returns:** +```json +{ + "export_id": "exp_123", + "status": "completed", + "documents_exported": 1523, + "file_size_bytes": 10485760, + "destination": "/exports/data.json" +} +``` + +--- + +## System Tools + +### health_check + +Check server health and status. + +**Parameters:** None + +**Returns:** +```json +{ + "status": "healthy", + "version": "0.1.0", + "uptime_seconds": 3600, + "transport": "http", + "dataset": { + "connected": true, + "document_count": 10523 + } +} +``` + +### list_tools + +List all available tools. + +**Parameters:** +```json +{ + "category": "string", // Filter by category + "include_schema": "boolean" // Include input schemas +} +``` + +**Returns:** +```json +{ + "tools": [ + { + "name": "search_documents", + "category": "search", + "description": "Search documents using text, vector, or hybrid search", + "input_schema": {...} + } + ], + "total_count": 43 +} +``` + +--- + +## Error Handling + +All tools return structured errors: + +```json +{ + "error": { + "type": "ValidationError", + "code": "INVALID_PARAMETER", + "message": "Parameter 'query' is required", + "details": { + "parameter": "query", + "provided": null, + "expected": "string" + } + } +} +``` + +Common error codes: +- `INVALID_PARAMETER` - Invalid input parameters +- `NOT_FOUND` - Resource not found +- `DUPLICATE_ID` - ID already exists +- `RATE_LIMIT_EXCEEDED` - Too many requests +- `UNAUTHORIZED` - Authentication failed +- `FORBIDDEN` - Insufficient permissions +- `INTERNAL_ERROR` - Server error + +## Rate Limits + +Default rate limits per tool category: + +| Category | Requests/Min | Burst | +|----------|-------------|-------| +| Search | 100 | 200 | +| Document | 50 | 100 | +| Collection | 30 | 60 | +| Import/Export | 10 | 20 | +| Analytics | 20 | 40 | +| System | 100 | 200 | + +## Next Steps + +- [Integration Examples](../cookbook/index.md) - Real-world usage +- [Error Reference](../reference/errors.md) - Complete error codes +- [Performance Guide](../guides/performance.md) - Optimization tips \ No newline at end of file diff --git a/docs/mcp/concepts/overview.md b/docs/mcp/concepts/overview.md new file mode 100644 index 0000000..f0c9e1e --- /dev/null +++ b/docs/mcp/concepts/overview.md @@ -0,0 +1,323 @@ +# Core Concepts + +Understanding the fundamental concepts of the ContextFrame MCP server will help you build more effective integrations and use the system to its full potential. + +## Model Context Protocol (MCP) + +### What is MCP? + +The Model Context Protocol is an open standard for enabling Large Language Models (LLMs) to interact with external systems in a structured, secure way. Think of it as a "USB interface" for AI - a standardized way to plug different tools and data sources into any AI system. + +### Key Principles + +1. **Tool-Based Interface**: All functionality exposed as discrete tools with well-defined inputs/outputs +2. **Transport Agnostic**: Works over HTTP, stdio, or other transports +3. **Security First**: Built-in authentication, authorization, and rate limiting +4. **Stateless Operations**: Each request is independent for easy scaling + +## Architecture Overview + +```mermaid +graph TB + subgraph "Client Layer" + A[AI Agent/LLM] + B[MCP Client Library] + C[Custom Applications] + end + + subgraph "MCP Server" + D[Transport Layer] + E[Security Middleware] + F[Tool Registry] + G[Request Handler] + H[Monitoring] + end + + subgraph "ContextFrame Core" + I[Dataset Manager] + J[Document Store] + K[Search Engine] + L[Vector Index] + end + + A --> B + B --> D + C --> D + D --> E + E --> F + F --> G + G --> I + E --> H + I --> J + I --> K + I --> L +``` + +## Core Components + +### 1. Transport Layer + +The transport layer handles communication between clients and the MCP server: + +- **HTTP Transport** (Primary): RESTful API with JSON payloads +- **Stdio Transport**: Direct process communication for local integrations +- **SSE Streaming**: Server-sent events for real-time updates + +Example HTTP request: +```json +POST /mcp/v1/tools/search_documents +{ + "params": { + "query": "machine learning", + "limit": 10 + } +} +``` + +### 2. Tool System + +Tools are the fundamental unit of functionality in MCP: + +```python +# Tool definition structure +{ + "name": "search_documents", + "description": "Search for documents using text or semantic search", + "input_schema": { + "type": "object", + "properties": { + "query": {"type": "string"}, + "limit": {"type": "integer", "default": 10} + }, + "required": ["query"] + } +} +``` + +Tool categories in ContextFrame: +- **Document Operations**: CRUD operations on documents +- **Search Tools**: Text, vector, and hybrid search +- **Collection Management**: Organize documents +- **Analytics**: Statistics and insights +- **Import/Export**: Bulk data operations +- **System Tools**: Health, monitoring, configuration + +### 3. Security Model + +Multi-layered security approach: + +```mermaid +graph LR + A[Request] --> B[Authentication] + B --> C[Authorization] + C --> D[Rate Limiting] + D --> E[Audit Logging] + E --> F[Tool Execution] +``` + +- **Authentication**: Verify who is making the request +- **Authorization**: Check if they can perform the action +- **Rate Limiting**: Prevent abuse and ensure fair usage +- **Audit Logging**: Track all operations for compliance + +### 4. Dataset Management + +ContextFrame uses Lance format for storage: + +``` +dataset.lance/ +├── _metadata/ +│ ├── schema.json +│ └── manifest.json +├── data/ +│ ├── 0.parquet +│ ├── 1.parquet +│ └── ... +└── _indices/ + ├── vector_index/ + └── text_index/ +``` + +Key features: +- **Columnar Storage**: Efficient for analytics +- **Version Control**: Track dataset changes +- **Zero-Copy Reads**: Memory-mapped for performance +- **Cloud Native**: Works with S3, GCS, Azure + +## Data Model + +### Documents + +The fundamental unit of data: + +```json +{ + "id": "doc_abc123", + "content": "Document content...", + "metadata": { + "title": "Document Title", + "created_at": "2024-01-01T00:00:00Z", + "tags": ["topic1", "topic2"], + "custom_field": "value" + }, + "embeddings": [0.1, 0.2, ...], // Optional vector + "dataset_id": "main", + "collection_id": "research" // Optional +} +``` + +### Collections + +Logical groupings of documents: + +```json +{ + "id": "coll_research", + "name": "Research Papers", + "description": "Academic research collection", + "metadata": { + "department": "AI Lab", + "access_level": "public" + }, + "document_count": 1523 +} +``` + +### Relationships + +Documents can have relationships: +- **Parent/Child**: Hierarchical structure +- **Related**: Semantic connections +- **References**: Citations or links + +## Request Flow + +Understanding how requests are processed: + +```mermaid +sequenceDiagram + participant Client + participant Transport + participant Security + participant Tool + participant Dataset + + Client->>Transport: MCP Request + Transport->>Security: Authenticate + Security->>Security: Authorize + Security->>Tool: Execute Tool + Tool->>Dataset: Query/Update + Dataset-->>Tool: Result + Tool-->>Security: Response + Security-->>Transport: Formatted Response + Transport-->>Client: MCP Response +``` + +## Key Patterns + +### 1. Batch Operations + +Efficient bulk processing: +```python +# Instead of multiple calls +for doc in documents: + client.document_create(doc) + +# Use batch operation +client.document_create_batch(documents) +``` + +### 2. Streaming Results + +For large result sets: +```python +# Stream search results +stream = client.search_documents_stream( + query="large dataset", + chunk_size=100 +) +for chunk in stream: + process(chunk) +``` + +### 3. Transactional Updates + +Atomic operations: +```python +# All succeed or all fail +with client.transaction() as tx: + tx.document_update(id1, content1) + tx.document_update(id2, content2) + tx.collection_add_documents(coll_id, [id1, id2]) +``` + +## Error Handling + +MCP uses structured error responses: + +```json +{ + "error": { + "code": "RATE_LIMIT_EXCEEDED", + "message": "Too many requests", + "details": { + "limit": 100, + "window": "1h", + "retry_after": 3600 + } + } +} +``` + +Error categories: +- **Client Errors** (4xx): Invalid requests, auth failures +- **Server Errors** (5xx): Internal errors, temporary failures +- **MCP Errors**: Protocol-specific errors + +## Performance Considerations + +### Caching + +Built-in caching for common operations: +- Search results cached for identical queries +- Collection metadata cached with TTL +- Document lookups use LRU cache + +### Connection Pooling + +HTTP transport uses connection pooling: +```python +# Reuse connections +client = MCPClient( + "http://localhost:8000", + pool_size=10, + pool_timeout=30 +) +``` + +### Pagination + +Efficient handling of large results: +```python +# Paginated queries +results = client.search_documents( + query="python", + limit=50, + offset=100 +) +``` + +## Best Practices + +1. **Use Appropriate Tools**: Choose the right tool for your use case +2. **Batch When Possible**: Reduce overhead with batch operations +3. **Handle Errors Gracefully**: Implement retry logic for transient failures +4. **Monitor Usage**: Track metrics to optimize performance +5. **Secure by Default**: Always use authentication in production + +## Next Steps + +- [API Reference](../api/tools.md) - Detailed tool documentation +- [Security Guide](../configuration/security.md) - Configure authentication +- [Integration Patterns](../guides/integration-patterns.md) - Common use cases +- [Performance Tuning](../guides/performance.md) - Optimization tips \ No newline at end of file diff --git a/docs/mcp/concepts/tools.md b/docs/mcp/concepts/tools.md new file mode 100644 index 0000000..e9ae014 --- /dev/null +++ b/docs/mcp/concepts/tools.md @@ -0,0 +1,354 @@ +# Understanding MCP Tools + +Tools are the fundamental building blocks of the MCP protocol. Each tool represents a specific capability that AI agents can use to interact with ContextFrame datasets. + +## What is a Tool? + +A tool in MCP is: +- A well-defined function with inputs and outputs +- Self-describing with JSON Schema +- Stateless and idempotent (where possible) +- Secured with authentication and authorization + +Think of tools as API endpoints specifically designed for AI consumption. + +## Tool Anatomy + +Every MCP tool has the following structure: + +```json +{ + "name": "search_documents", + "description": "Search documents using text, vector, or hybrid search", + "inputSchema": { + "type": "object", + "properties": { + "query": { + "type": "string", + "description": "Search query text" + }, + "search_type": { + "type": "string", + "enum": ["text", "vector", "hybrid"], + "default": "hybrid" + }, + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 100, + "default": 10 + } + }, + "required": ["query"] + } +} +``` + +## Tool Categories + +### 1. Document Operations + +Core CRUD operations for documents: + +| Tool | Purpose | Example Use | +|------|---------|-------------| +| `document_create` | Create new document | Adding research papers | +| `document_get` | Retrieve by ID | Fetching specific doc | +| `document_update` | Modify existing | Updating metadata | +| `document_delete` | Remove document | Cleanup operations | +| `document_exists` | Check existence | Validation logic | + +### 2. Search Tools + +Various search capabilities: + +| Tool | Purpose | Search Type | +|------|---------|-------------| +| `search_documents` | General search | Text/Vector/Hybrid | +| `search_similar` | Find similar docs | Vector similarity | +| `search_by_metadata` | Filter by attributes | SQL-like queries | +| `search_within_collection` | Scoped search | Collection-specific | + +### 3. Collection Management + +Organize documents into groups: + +| Tool | Purpose | Typical Use | +|------|---------|-------------| +| `collection_create` | New collection | Project organization | +| `collection_list` | List all | Discovery | +| `collection_add_documents` | Add to collection | Categorization | +| `collection_remove_documents` | Remove from | Reorganization | +| `collection_stats` | Get statistics | Analytics | + +### 4. Analytics Tools + +Insights and statistics: + +| Tool | Purpose | Returns | +|------|---------|---------| +| `dataset_stats` | Overall statistics | Counts, size, etc | +| `search_analytics` | Search patterns | Popular queries | +| `usage_metrics` | Usage tracking | API calls, costs | +| `document_analytics` | Document insights | Relationships, clusters | + +### 5. Import/Export Tools + +Bulk data operations: + +| Tool | Purpose | Formats | +|------|---------|---------| +| `import_documents` | Bulk import | JSON, CSV, Parquet | +| `export_documents` | Bulk export | Multiple formats | +| `import_from_url` | Remote import | Web sources | +| `export_to_cloud` | Cloud export | S3, GCS, Azure | + +### 6. System Tools + +Management and monitoring: + +| Tool | Purpose | Usage | +|------|---------|-------| +| `health_check` | Server status | Monitoring | +| `list_tools` | Available tools | Discovery | +| `validate_dataset` | Check integrity | Maintenance | +| `clear_cache` | Cache management | Performance | + +## Tool Execution Flow + +```mermaid +graph TD + A[Client Request] --> B{Validate Schema} + B -->|Valid| C[Check Auth] + B -->|Invalid| G[Schema Error] + C -->|Authorized| D[Check Rate Limit] + C -->|Unauthorized| H[Auth Error] + D -->|Within Limit| E[Execute Tool] + D -->|Exceeded| I[Rate Limit Error] + E --> F[Return Result] + E --> J[Log Operation] +``` + +## Tool Patterns + +### 1. Idempotent Operations + +Many tools are idempotent - calling them multiple times has the same effect: + +```python +# Creating with explicit ID is idempotent +client.document_create( + id="doc_123", + content="...", + upsert=True # Create or update +) +``` + +### 2. Batch Operations + +Tools that support batch processing: + +```python +# Single operation +for doc in documents: + client.document_create(doc) + +# Batch operation (more efficient) +client.document_create_batch(documents) +``` + +### 3. Streaming Results + +Tools that can stream large results: + +```python +# Regular search (all results at once) +results = client.search_documents(query="AI", limit=1000) + +# Streaming search (memory efficient) +for batch in client.search_documents_stream(query="AI", batch_size=100): + process_batch(batch) +``` + +### 4. Transactional Tools + +Some tools support transactions: + +```python +# Atomic operations +with client.transaction() as tx: + doc_id = tx.document_create(content="...") + tx.collection_add_documents("research", [doc_id]) + # All operations succeed or all fail +``` + +## Input Validation + +All tool inputs are validated against their schema: + +```python +# This will fail validation +client.search_documents( + query="test", + limit=1000 # Max is 100 +) +# Error: Invalid input: limit must be <= 100 + +# Proper usage +client.search_documents( + query="test", + limit=100 # Within bounds +) +``` + +## Output Formats + +Tools return structured responses: + +```json +{ + "result": { + "documents": [...], + "total_count": 156, + "execution_time_ms": 23 + }, + "metadata": { + "tool": "search_documents", + "version": "1.0", + "cached": false + } +} +``` + +## Error Handling + +Tools provide detailed error information: + +```json +{ + "error": { + "type": "ToolExecutionError", + "tool": "document_create", + "code": "DUPLICATE_ID", + "message": "Document with ID 'doc_123' already exists", + "suggestion": "Use upsert=true to update existing document" + } +} +``` + +## Tool Discovery + +Clients can discover available tools: + +```python +# List all tools +tools = client.list_tools() + +# Get specific tool info +tool_info = client.get_tool_info("search_documents") +print(tool_info["description"]) +print(tool_info["inputSchema"]) +``` + +## Custom Tools + +While ContextFrame provides 43 built-in tools, you can extend with custom tools: + +```python +@mcp_tool( + name="summarize_collection", + description="Generate summary of all documents in a collection" +) +def summarize_collection(collection_id: str, max_length: int = 500): + # Custom implementation + documents = get_collection_documents(collection_id) + return generate_summary(documents, max_length) +``` + +## Tool Composition + +Tools can be composed for complex operations: + +```python +# Find and export related documents +def export_related_documents(doc_id: str, output_path: str): + # 1. Get the document + doc = client.document_get(doc_id) + + # 2. Find similar documents + similar = client.search_similar( + document_id=doc_id, + limit=50 + ) + + # 3. Export all to file + doc_ids = [doc_id] + [d["id"] for d in similar] + client.export_documents( + document_ids=doc_ids, + format="json", + output_path=output_path + ) +``` + +## Performance Tips + +### 1. Use Appropriate Tools + +Choose the most specific tool for your needs: +```python +# Less efficient +all_docs = client.document_list() +filtered = [d for d in all_docs if d["metadata"]["type"] == "paper"] + +# More efficient +filtered = client.search_by_metadata( + filters={"type": "paper"} +) +``` + +### 2. Leverage Caching + +Some tools cache results: +```python +# First call: hits the database +results1 = client.collection_stats("research") + +# Second call within TTL: returns cached +results2 = client.collection_stats("research") +``` + +### 3. Batch When Possible + +Reduce overhead: +```python +# Instead of 100 API calls +for id in document_ids: + client.document_delete(id) + +# Make 1 API call +client.document_delete_batch(document_ids) +``` + +## Security Considerations + +Tools respect security boundaries: + +1. **Authentication**: Every tool requires valid credentials +2. **Authorization**: Tools check permissions before execution +3. **Rate Limiting**: Prevents abuse and ensures fair usage +4. **Audit Logging**: All tool executions are logged + +## Best Practices + +1. **Read Tool Documentation**: Understand inputs, outputs, and limitations +2. **Handle Errors**: Implement proper error handling +3. **Use Batch Operations**: When processing multiple items +4. **Monitor Usage**: Track tool execution metrics +5. **Cache Results**: When appropriate for your use case + +## Next Steps + +- [API Reference](../api/tools.md) - Detailed documentation for each tool +- [Security Configuration](../configuration/security.md) - Set up tool permissions +- [Integration Examples](../cookbook/index.md) - Real-world tool usage +- [Performance Guide](../guides/performance.md) - Optimize tool usage \ No newline at end of file diff --git a/docs/mcp/concepts/transport.md b/docs/mcp/concepts/transport.md new file mode 100644 index 0000000..f89ffb1 --- /dev/null +++ b/docs/mcp/concepts/transport.md @@ -0,0 +1,456 @@ +# Transport Layers + +The ContextFrame MCP server supports multiple transport mechanisms to accommodate different deployment scenarios and integration patterns. This guide explains each transport option and when to use them. + +## Transport Overview + +```mermaid +graph LR + subgraph "Clients" + A[Web Apps] + B[AI Agents] + C[CLI Tools] + D[Native Apps] + end + + subgraph "Transports" + E[HTTP/REST] + F[Stdio] + G[SSE Streaming] + end + + subgraph "MCP Server" + H[Request Handler] + end + + A --> E + B --> E + B --> F + C --> F + D --> E + E --> H + F --> H + G --> H +``` + +## HTTP Transport (Primary) + +HTTP is the recommended transport for most use cases, providing a RESTful API interface. + +### Features + +- **Universal Compatibility**: Works with any HTTP client +- **Stateless Design**: Easy to scale horizontally +- **Standard Security**: TLS, authentication headers +- **Load Balancing**: Works with standard infrastructure +- **Firewall Friendly**: Uses standard ports + +### Configuration + +```yaml +transport: + type: http + config: + host: 0.0.0.0 + port: 8000 + cors: + enabled: true + origins: ["*"] + ssl: + enabled: true + cert_file: /path/to/cert.pem + key_file: /path/to/key.pem +``` + +### Request Format + +```http +POST /mcp/v1/tools/search_documents HTTP/1.1 +Host: api.contextframe.com +Content-Type: application/json +Authorization: Bearer your-api-key + +{ + "params": { + "query": "machine learning", + "limit": 10 + } +} +``` + +### Response Format + +```http +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "result": { + "documents": [...], + "total_count": 42 + }, + "metadata": { + "execution_time_ms": 15 + } +} +``` + +### When to Use HTTP + +- **Production deployments**: Best for scalable, secure deployments +- **Multi-tenant systems**: Easy to isolate tenants +- **Cloud environments**: Works with load balancers, API gateways +- **Remote access**: Accessible over the internet +- **Monitoring**: Integrates with standard tools + +## Stdio Transport + +Stdio (standard input/output) transport enables direct process communication, ideal for local integrations. + +### Features + +- **Zero Network Overhead**: Direct process communication +- **Simple Integration**: No network configuration needed +- **Secure by Default**: No network exposure +- **Low Latency**: Minimal overhead + +### Configuration + +```yaml +transport: + type: stdio + config: + buffer_size: 8192 + timeout_ms: 30000 +``` + +### Starting Stdio Mode + +```bash +# Basic stdio mode +contextframe-mcp stdio + +# With specific dataset +contextframe-mcp stdio --dataset-path /data/my-dataset.lance + +# With configuration +contextframe-mcp stdio --config mcp-config.yaml +``` + +### Message Format + +Input (stdin): +```json +{ + "jsonrpc": "2.0", + "method": "tools/search_documents", + "params": { + "query": "quantum computing", + "limit": 5 + }, + "id": 1 +} +``` + +Output (stdout): +```json +{ + "jsonrpc": "2.0", + "result": { + "documents": [...], + "total_count": 12 + }, + "id": 1 +} +``` + +### Integration Example + +```python +import subprocess +import json + +# Start MCP server in stdio mode +proc = subprocess.Popen( + ["contextframe-mcp", "stdio"], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True +) + +# Send request +request = { + "jsonrpc": "2.0", + "method": "tools/document_create", + "params": { + "content": "Hello, World!", + "metadata": {"type": "greeting"} + }, + "id": 1 +} + +proc.stdin.write(json.dumps(request) + "\n") +proc.stdin.flush() + +# Read response +response = json.loads(proc.stdout.readline()) +print(response["result"]) +``` + +### When to Use Stdio + +- **Local AI agents**: Claude Desktop, local LLMs +- **Development**: Quick testing without network setup +- **Embedded systems**: Minimal resource usage +- **Security-critical**: No network exposure +- **Single-user apps**: Desktop applications + +## SSE Streaming Transport + +Server-Sent Events enable real-time streaming of results, useful for long-running operations. + +### Features + +- **Real-time Updates**: Stream results as they're generated +- **Resumable**: Client can reconnect and resume +- **Efficient**: Lower overhead than polling +- **Standard Protocol**: Works with EventSource API + +### Configuration + +```yaml +transport: + sse: + enabled: true + endpoint: /mcp/v1/stream + heartbeat_interval: 30 + max_connections: 1000 +``` + +### Client Example + +```javascript +const eventSource = new EventSource('/mcp/v1/stream/search?query=AI'); + +eventSource.onmessage = (event) => { + const result = JSON.parse(event.data); + console.log('Received:', result); +}; + +eventSource.onerror = (error) => { + console.error('Stream error:', error); +}; +``` + +### Stream Format + +``` +event: document +data: {"id": "doc_123", "content": "...", "score": 0.95} + +event: document +data: {"id": "doc_124", "content": "...", "score": 0.92} + +event: complete +data: {"total_count": 150, "execution_time_ms": 234} +``` + +### When to Use SSE + +- **Large result sets**: Stream results as they're found +- **Real-time monitoring**: Live metrics and updates +- **Progress tracking**: Long-running operations +- **Incremental results**: Show results as available + +## Transport Selection Guide + +| Use Case | Recommended Transport | Why | +|----------|---------------------|-----| +| Production API | HTTP | Scalable, secure, standard | +| Claude Desktop | Stdio | Direct integration | +| Web Application | HTTP | Browser compatible | +| Batch Processing | HTTP | Better error handling | +| Real-time Dashboard | SSE | Live updates | +| Embedded System | Stdio | Minimal resources | +| Microservices | HTTP | Service mesh compatible | + +## Security Considerations + +### HTTP Security + +```yaml +security: + http: + tls: + enabled: true + min_version: "1.2" + headers: + - "X-API-Key" + - "Authorization" + cors: + allowed_origins: ["https://app.example.com"] + allowed_methods: ["POST", "GET"] +``` + +### Stdio Security + +- Process isolation +- User permissions +- No network exposure +- Audit via process monitoring + +### SSE Security + +- Same-origin policy +- Authentication tokens +- Connection limits +- Rate limiting + +## Performance Comparison + +| Transport | Latency | Throughput | CPU Usage | Memory | +|-----------|---------|------------|-----------|---------| +| HTTP | ~1-5ms | High | Moderate | Moderate | +| Stdio | <1ms | Very High | Low | Low | +| SSE | ~1-2ms | Moderate | Low | Low | + +## Advanced Configuration + +### HTTP with Connection Pooling + +```python +from contextframe.mcp import MCPClient + +client = MCPClient( + base_url="http://localhost:8000", + transport_config={ + "pool_connections": 10, + "pool_maxsize": 50, + "max_retries": 3, + "timeout": 30 + } +) +``` + +### Stdio with Custom Protocol + +```python +class CustomStdioTransport(StdioTransport): + def encode_message(self, message): + # Custom encoding + return msgpack.packb(message) + + def decode_message(self, data): + # Custom decoding + return msgpack.unpackb(data) +``` + +### SSE with Reconnection + +```javascript +class ReconnectingEventSource { + constructor(url, options = {}) { + this.url = url; + this.reconnectInterval = options.reconnectInterval || 3000; + this.connect(); + } + + connect() { + this.eventSource = new EventSource(this.url); + + this.eventSource.onerror = () => { + this.eventSource.close(); + setTimeout(() => this.connect(), this.reconnectInterval); + }; + } +} +``` + +## Monitoring Transport Health + +### HTTP Metrics + +```python +# Prometheus metrics +mcp_http_requests_total +mcp_http_request_duration_seconds +mcp_http_request_size_bytes +mcp_http_response_size_bytes +``` + +### Stdio Metrics + +```python +# Process metrics +mcp_stdio_messages_total +mcp_stdio_message_size_bytes +mcp_stdio_processing_time_seconds +``` + +### SSE Metrics + +```python +# Streaming metrics +mcp_sse_connections_active +mcp_sse_messages_sent_total +mcp_sse_connection_duration_seconds +``` + +## Troubleshooting + +### HTTP Issues + +```bash +# Test connectivity +curl -X GET http://localhost:8000/health + +# Check TLS +openssl s_client -connect api.contextframe.com:443 + +# Debug headers +curl -v -X POST http://localhost:8000/mcp/v1/tools/list_tools +``` + +### Stdio Issues + +```bash +# Test stdio mode +echo '{"jsonrpc":"2.0","method":"health_check","id":1}' | contextframe-mcp stdio + +# Check process +ps aux | grep contextframe-mcp + +# Monitor I/O +strace -e read,write contextframe-mcp stdio +``` + +### SSE Issues + +```javascript +// Debug SSE connection +const eventSource = new EventSource('/mcp/v1/stream/test'); + +eventSource.addEventListener('open', () => { + console.log('Connection opened'); +}); + +eventSource.addEventListener('error', (e) => { + console.log('Connection error:', e); +}); +``` + +## Best Practices + +1. **Choose the Right Transport**: Match transport to use case +2. **Implement Retry Logic**: Handle transient failures +3. **Monitor Performance**: Track latency and throughput +4. **Secure All Transports**: Even stdio needs access control +5. **Test Failover**: Ensure graceful degradation + +## Next Steps + +- [HTTP API Reference](../api/http.md) - Detailed HTTP endpoints +- [Stdio Protocol](../reference/stdio-protocol.md) - Message format spec +- [SSE Integration](../guides/sse-streaming.md) - Streaming guide +- [Security Configuration](../configuration/security.md) - Secure your transport \ No newline at end of file diff --git a/docs/mcp/configuration/index.md b/docs/mcp/configuration/index.md new file mode 100644 index 0000000..8523e33 --- /dev/null +++ b/docs/mcp/configuration/index.md @@ -0,0 +1,501 @@ +# Configuration Guide + +The ContextFrame MCP server offers flexible configuration options to suit various deployment scenarios, from development to production. This guide covers all configuration aspects. + +## Configuration Overview + +Configuration can be provided through: +1. **Configuration files** (YAML/JSON) +2. **Environment variables** +3. **Command-line arguments** +4. **Default values** + +Priority order (highest to lowest): +1. Command-line arguments +2. Environment variables +3. Configuration file +4. Default values + +## Configuration Files + +### Basic Configuration + +Create a `contextframe-mcp.yaml` file: + +```yaml +# Server configuration +server: + host: 0.0.0.0 + port: 8000 + workers: 4 + +# Dataset configuration +dataset: + path: /data/contextframe/dataset.lance + create_if_missing: true + +# Transport configuration +transport: + type: http # or "stdio" + http: + cors: + enabled: true + origins: ["http://localhost:3000"] + +# Security configuration +security: + enabled: true + auth: + type: api_key + api_key: ${CONTEXTFRAME_API_KEY} + +# Monitoring configuration +monitoring: + enabled: true + level: INFO + metrics: + enabled: true + export_interval: 60 +``` + +### Using the Configuration + +```bash +# Specify config file +contextframe-mcp serve --config /path/to/config.yaml + +# Or use default location +# Looks for: ./contextframe-mcp.yaml, ~/.contextframe/mcp.yaml +contextframe-mcp serve +``` + +## Environment Variables + +All configuration options can be set via environment variables: + +```bash +# Server settings +export CONTEXTFRAME_HOST=0.0.0.0 +export CONTEXTFRAME_PORT=8000 +export CONTEXTFRAME_WORKERS=4 + +# Dataset settings +export CONTEXTFRAME_DATASET_PATH=/data/dataset.lance +export CONTEXTFRAME_DATASET_CREATE_IF_MISSING=true + +# Security settings +export CONTEXTFRAME_SECURITY_ENABLED=true +export CONTEXTFRAME_API_KEY=your-secret-key + +# Monitoring settings +export CONTEXTFRAME_MONITORING_ENABLED=true +export CONTEXTFRAME_LOG_LEVEL=INFO +``` + +### Environment Variable Naming + +- Prefix: `CONTEXTFRAME_` +- Nested values use underscores: `CONTEXTFRAME_SECURITY_AUTH_TYPE` +- Lists use comma separation: `CONTEXTFRAME_CORS_ORIGINS=http://localhost:3000,https://app.example.com` + +## Configuration Sections + +### Server Configuration + +Basic server settings: + +```yaml +server: + # Network binding + host: 0.0.0.0 # Listen address + port: 8000 # Listen port + + # Performance + workers: 4 # Number of worker processes + worker_class: uvicorn # Worker type + + # Timeouts + timeout: 30 # Request timeout (seconds) + keepalive: 5 # Keep-alive timeout + + # SSL/TLS + ssl: + enabled: false + cert_file: /path/to/cert.pem + key_file: /path/to/key.pem + + # Development + reload: false # Auto-reload on changes + debug: false # Debug mode +``` + +### Dataset Configuration + +Lance dataset settings: + +```yaml +dataset: + # Primary dataset + path: /data/dataset.lance + create_if_missing: true + + # Storage backend + storage: + type: local # local, s3, gcs, azure + options: + # S3 example + # bucket: my-bucket + # region: us-east-1 + # access_key_id: ${AWS_ACCESS_KEY_ID} + # secret_access_key: ${AWS_SECRET_ACCESS_KEY} + + # Performance + cache: + enabled: true + size_mb: 1024 # Cache size in MB + ttl_seconds: 3600 # Cache TTL + + # Indexing + index: + vector: + enabled: true + dimensions: 768 # Embedding dimensions + metric: cosine # cosine, euclidean, dot + text: + enabled: true + analyzer: standard # standard, english, multilingual +``` + +### Security Configuration + +Detailed security guide at [Security Configuration](security.md): + +```yaml +security: + enabled: true + + # Authentication + auth: + type: multiple # api_key, bearer, basic, oauth2, multiple + providers: + - type: api_key + header: X-API-Key + keys: + - ${API_KEY_1} + - ${API_KEY_2} + - type: bearer + jwt: + secret: ${JWT_SECRET} + algorithm: HS256 + + # Authorization + authorization: + enabled: true + default_role: reader + roles: + reader: + - tools:read + - documents:read + writer: + - tools:* + - documents:* + - collections:* + + # Rate limiting + rate_limiting: + enabled: true + default: + requests: 100 + window: 60 # seconds + by_endpoint: + search_documents: + requests: 50 + window: 60 +``` + +### Monitoring Configuration + +Detailed monitoring guide at [Monitoring Setup](monitoring.md): + +```yaml +monitoring: + # Logging + logging: + level: INFO # DEBUG, INFO, WARNING, ERROR + format: json # json, text + output: stdout # stdout, file + file: + path: /var/log/contextframe/mcp.log + rotation: daily + retention: 7 # days + + # Metrics + metrics: + enabled: true + backend: prometheus # prometheus, statsd + export_interval: 60 # seconds + endpoint: /metrics + + # Tracing + tracing: + enabled: false + backend: jaeger # jaeger, zipkin + sample_rate: 0.1 # 10% sampling + endpoint: http://localhost:14268 + + # Health checks + health: + enabled: true + endpoint: /health + checks: + - dataset + - memory + - disk +``` + +### Transport Configuration + +Transport-specific settings: + +```yaml +transport: + type: http # http, stdio + + # HTTP transport + http: + # CORS + cors: + enabled: true + origins: ["*"] # Or specific origins + methods: ["GET", "POST"] + headers: ["Content-Type", "Authorization"] + + # Compression + compression: + enabled: true + min_size: 1024 # Minimum size to compress + + # Request limits + limits: + max_request_size: 10485760 # 10MB + max_header_size: 8192 # 8KB + + # Stdio transport + stdio: + buffer_size: 8192 + encoding: utf-8 + line_delimiter: "\n" +``` + +## Advanced Configuration + +### Multi-Environment Setup + +Use different configs for each environment: + +```yaml +# config/development.yaml +extends: base.yaml +server: + debug: true + reload: true +security: + enabled: false + +# config/production.yaml +extends: base.yaml +server: + workers: 8 +security: + enabled: true + auth: + type: oauth2 +``` + +### Dynamic Configuration + +Load configuration from external sources: + +```yaml +# Load from AWS Parameter Store +config_source: + type: aws_parameter_store + prefix: /contextframe/mcp/ + region: us-east-1 + +# Load from Kubernetes ConfigMap +config_source: + type: kubernetes + configmap: contextframe-mcp-config + namespace: default +``` + +### Feature Flags + +Enable/disable features: + +```yaml +features: + # Experimental features + experimental: + streaming_search: false + graph_relationships: false + + # Beta features + beta: + advanced_analytics: true + custom_embeddings: true +``` + +## Configuration Validation + +The server validates configuration on startup: + +```bash +# Validate configuration without starting +contextframe-mcp validate-config --config config.yaml + +# Show effective configuration +contextframe-mcp show-config --config config.yaml +``` + +## Common Configurations + +### Development Setup + +```yaml +server: + host: localhost + port: 8000 + reload: true + debug: true + +dataset: + path: ./dev-dataset.lance + create_if_missing: true + +security: + enabled: false + +monitoring: + logging: + level: DEBUG +``` + +### Production Setup + +```yaml +server: + host: 0.0.0.0 + port: 8000 + workers: ${WORKERS:-4} + ssl: + enabled: true + cert_file: /etc/ssl/certs/server.crt + key_file: /etc/ssl/private/server.key + +dataset: + path: s3://my-bucket/prod-dataset.lance + storage: + type: s3 + options: + region: ${AWS_REGION} + +security: + enabled: true + auth: + type: oauth2 + oauth2: + issuer: https://auth.example.com + audience: contextframe-mcp + +monitoring: + metrics: + enabled: true + tracing: + enabled: true + sample_rate: 0.1 +``` + +### Docker Setup + +```yaml +server: + host: 0.0.0.0 # Important for Docker + port: ${PORT:-8000} + +dataset: + path: /data/dataset.lance + +# Use environment variables for secrets +security: + auth: + api_key: ${API_KEY} +``` + +## Configuration Best Practices + +1. **Use Environment Variables for Secrets** + ```yaml + api_key: ${API_KEY} # Good + api_key: "hardcoded-key" # Bad + ``` + +2. **Separate Configs by Environment** + - `config/base.yaml` - Shared settings + - `config/dev.yaml` - Development + - `config/prod.yaml` - Production + +3. **Version Control Configuration** + - Check in configuration files + - Never commit secrets + - Use `.gitignore` for local overrides + +4. **Validate Before Deployment** + ```bash + contextframe-mcp validate-config --config prod.yaml + ``` + +5. **Monitor Configuration Changes** + - Log configuration on startup + - Track configuration in monitoring + - Alert on configuration errors + +## Troubleshooting + +### Configuration Not Loading + +```bash +# Check configuration search paths +contextframe-mcp show-config-paths + +# Validate YAML syntax +yamllint contextframe-mcp.yaml + +# Check environment variables +env | grep CONTEXTFRAME_ +``` + +### Invalid Configuration + +```bash +# Show validation errors +contextframe-mcp validate-config --config config.yaml --verbose + +# Test with minimal config +contextframe-mcp serve --config minimal.yaml +``` + +### Performance Issues + +Check these settings: +- `workers`: Match CPU cores +- `cache.size_mb`: Increase for better performance +- `timeout`: Adjust for slow operations + +## Next Steps + +- [Security Configuration](security.md) - Detailed security setup +- [Monitoring Setup](monitoring.md) - Configure metrics and logging +- [Production Deployment](../guides/production-deployment.md) - Best practices +- [Performance Tuning](../guides/performance.md) - Optimization guide \ No newline at end of file diff --git a/docs/mcp/configuration/monitoring.md b/docs/mcp/configuration/monitoring.md new file mode 100644 index 0000000..59bdd9f --- /dev/null +++ b/docs/mcp/configuration/monitoring.md @@ -0,0 +1,735 @@ +# Monitoring Setup Guide + +Comprehensive monitoring is essential for operating the ContextFrame MCP server in production. This guide covers logging, metrics, tracing, and alerting configuration. + +## Monitoring Overview + +The MCP server provides built-in monitoring capabilities with minimal overhead: + +```mermaid +graph LR + subgraph "MCP Server" + A[Request Handler] + B[Metrics Collector] + C[Logger] + D[Tracer] + end + + subgraph "Backends" + E[Prometheus] + F[Elasticsearch] + G[Jaeger] + H[CloudWatch] + end + + A --> B + A --> C + A --> D + B --> E + C --> F + D --> G + B --> H +``` + +## Logging Configuration + +### Basic Logging + +```yaml +monitoring: + logging: + level: INFO # DEBUG, INFO, WARNING, ERROR + format: json # json, text, structured + + # Console output + console: + enabled: true + colorize: true # Color coding for levels + + # File output + file: + enabled: true + path: /var/log/contextframe/mcp.log + rotation: + type: size # size, time, both + max_size: 100MB + max_files: 10 + compress: true +``` + +### Structured Logging + +JSON format for easy parsing: + +```json +{ + "timestamp": "2024-01-15T10:30:45.123Z", + "level": "INFO", + "message": "Document created successfully", + "fields": { + "request_id": "req_abc123", + "user_id": "user_456", + "tool": "document_create", + "document_id": "doc_789", + "duration_ms": 45 + }, + "context": { + "service": "mcp-server", + "version": "0.1.0", + "environment": "production" + } +} +``` + +### Log Levels and Categories + +```yaml +monitoring: + logging: + # Global level + level: INFO + + # Per-component levels + levels: + contextframe.mcp.server: INFO + contextframe.mcp.auth: DEBUG + contextframe.mcp.tools: INFO + contextframe.lance: WARNING + + # Categories to include/exclude + categories: + include: + - security + - performance + - errors + exclude: + - health_checks + - metrics_export +``` + +### Log Aggregation + +Send logs to centralized systems: + +```yaml +monitoring: + logging: + aggregation: + # Elasticsearch/ELK + elasticsearch: + enabled: true + hosts: + - http://elasticsearch:9200 + index: contextframe-mcp + + # AWS CloudWatch + cloudwatch: + enabled: true + region: us-east-1 + log_group: /aws/contextframe/mcp + + # Splunk + splunk: + enabled: true + endpoint: https://splunk.example.com:8088 + token: ${SPLUNK_HEC_TOKEN} + + # Fluentd/Fluent Bit + fluentd: + enabled: true + host: fluentd + port: 24224 +``` + +## Metrics Collection + +### Prometheus Metrics + +Built-in Prometheus integration: + +```yaml +monitoring: + metrics: + enabled: true + backend: prometheus + + # Metrics endpoint + endpoint: + path: /metrics + port: 9090 # Separate port for metrics + auth: false # No auth for scraping + + # What to collect + collectors: + - request_duration + - request_size + - response_size + - active_connections + - tool_usage + - error_rates + - cache_hits + - dataset_operations + + # Custom labels + labels: + service: mcp-server + environment: ${ENVIRONMENT} + region: ${AWS_REGION} +``` + +### Available Metrics + +Core metrics exposed: + +```prometheus +# HTTP Metrics +mcp_http_requests_total{method="POST",endpoint="/mcp/v1/tools/search_documents",status="200"} +mcp_http_request_duration_seconds{quantile="0.99"} +mcp_http_request_size_bytes_sum +mcp_http_response_size_bytes_sum + +# Tool Metrics +mcp_tool_executions_total{tool="search_documents",status="success"} +mcp_tool_duration_seconds{tool="search_documents",quantile="0.95"} +mcp_tool_errors_total{tool="document_create",error="validation_failed"} + +# Dataset Metrics +mcp_dataset_documents_total +mcp_dataset_size_bytes +mcp_dataset_operations_total{operation="read"} + +# Search Metrics +mcp_search_queries_total{type="hybrid"} +mcp_search_latency_seconds{type="vector",quantile="0.99"} +mcp_search_results_returned_total + +# Cache Metrics +mcp_cache_hits_total{cache="document"} +mcp_cache_misses_total{cache="search"} +mcp_cache_evictions_total + +# System Metrics +mcp_active_connections +mcp_memory_usage_bytes +mcp_cpu_usage_percent +``` + +### StatsD Integration + +For existing StatsD infrastructure: + +```yaml +monitoring: + metrics: + backend: statsd + statsd: + host: statsd.example.com + port: 8125 + prefix: contextframe.mcp + + # Metric types + timers: + - request_duration + - tool_execution_time + counters: + - requests_total + - errors_total + gauges: + - active_connections + - queue_size +``` + +### Custom Metrics + +Define application-specific metrics: + +```yaml +monitoring: + metrics: + custom: + # Document complexity metric + - name: document_complexity_score + type: histogram + buckets: [0.1, 0.5, 1.0, 2.0, 5.0] + labels: ["collection", "type"] + + # Cost tracking + - name: operation_cost_units + type: counter + labels: ["tool", "user", "tenant"] + + # SLA tracking + - name: sla_compliance_rate + type: gauge + labels: ["service_level", "customer"] +``` + +## Distributed Tracing + +### OpenTelemetry Setup + +```yaml +monitoring: + tracing: + enabled: true + backend: opentelemetry + + # OTLP Exporter + otlp: + endpoint: http://otel-collector:4317 + protocol: grpc # grpc or http + + # Sampling + sampling: + type: probability # probability, rate_limiting + rate: 0.1 # 10% sampling + + # Trace enrichment + attributes: + service.name: mcp-server + service.version: ${VERSION} + deployment.environment: ${ENVIRONMENT} +``` + +### Jaeger Integration + +```yaml +monitoring: + tracing: + backend: jaeger + jaeger: + agent_host: jaeger-agent + agent_port: 6831 + + # Or collector endpoint + collector_endpoint: http://jaeger:14268/api/traces + + # Service name + service_name: contextframe-mcp + + # Tags + tags: + environment: production + region: us-east-1 +``` + +### Trace Context + +Automatic trace context propagation: + +```yaml +monitoring: + tracing: + propagation: + # Formats to accept/inject + formats: + - tracecontext # W3C Trace Context + - baggage # W3C Baggage + - b3 # Zipkin B3 + - jaeger # Uber Jaeger + + # What to trace + instrumentation: + - http_server + - http_client + - database + - cache + - tools +``` + +## Health Checks + +### Basic Health Endpoint + +```yaml +monitoring: + health: + enabled: true + endpoint: /health + + # Checks to perform + checks: + # Dataset connectivity + dataset: + enabled: true + timeout: 5s + + # Memory usage + memory: + enabled: true + threshold: 90 # Percent + + # Disk space + disk: + enabled: true + threshold: 85 # Percent + path: /data + + # Custom checks + custom: + - name: embedding_service + type: http + url: http://embeddings:8080/health + timeout: 2s +``` + +### Health Response Format + +```json +{ + "status": "healthy", // healthy, degraded, unhealthy + "timestamp": "2024-01-15T10:30:00Z", + "version": "0.1.0", + "checks": { + "dataset": { + "status": "healthy", + "message": "Connected to dataset", + "duration_ms": 5 + }, + "memory": { + "status": "healthy", + "usage_percent": 65, + "available_mb": 3584 + }, + "disk": { + "status": "warning", + "usage_percent": 82, + "available_gb": 45 + } + } +} +``` + +### Readiness vs Liveness + +```yaml +monitoring: + health: + # Liveness - is the service running? + liveness: + endpoint: /health/live + checks: [process, memory] + + # Readiness - can it serve traffic? + readiness: + endpoint: /health/ready + checks: [dataset, cache, auth] + + # Startup - initial startup check + startup: + endpoint: /health/startup + initial_delay: 10s + period: 5s + failure_threshold: 3 +``` + +## Performance Monitoring + +### Request Performance + +```yaml +monitoring: + performance: + # Track slow requests + slow_requests: + enabled: true + threshold_ms: 1000 + log_details: true + + # Request profiling + profiling: + enabled: false # Enable for debugging + sample_rate: 0.01 # 1% of requests + + # Performance budgets + budgets: + search_documents: + p50: 50ms + p95: 200ms + p99: 500ms + document_create: + p50: 100ms + p95: 300ms + p99: 1000ms +``` + +### Resource Monitoring + +```yaml +monitoring: + resources: + # CPU monitoring + cpu: + enabled: true + alert_threshold: 80 # Percent + + # Memory monitoring + memory: + enabled: true + alert_threshold: 85 # Percent + track_allocations: false # Detailed tracking + + # Connection pools + connections: + track_pools: true + alert_on_exhaustion: true + + # Thread pools + threads: + track_usage: true + dump_on_deadlock: true +``` + +## Alerting Configuration + +### Alert Rules + +```yaml +monitoring: + alerting: + enabled: true + + # Alert destinations + destinations: + - type: webhook + url: https://alerts.example.com/webhook + + - type: email + smtp_host: smtp.example.com + recipients: + - ops@example.com + + - type: pagerduty + integration_key: ${PAGERDUTY_KEY} + + # Alert rules + rules: + # High error rate + - name: high_error_rate + condition: | + rate(mcp_tool_errors_total[5m]) > 0.05 + severity: warning + notify: [email] + + # Service down + - name: service_down + condition: | + up{job="mcp-server"} == 0 + severity: critical + notify: [pagerduty] + + # Slow response times + - name: slow_responses + condition: | + histogram_quantile(0.95, mcp_http_request_duration_seconds) > 1.0 + severity: warning + notify: [webhook] +``` + +### Alert Aggregation + +```yaml +monitoring: + alerting: + aggregation: + # Group related alerts + grouping: + - by: [service, environment] + wait: 30s + interval: 5m + + # Silence rules + silences: + - name: maintenance_window + schedule: "0 2 * * SUN" # Sunday 2 AM + duration: 2h + + # Deduplication + deduplication: + enabled: true + window: 5m +``` + +## Dashboard Configuration + +### Grafana Integration + +```yaml +monitoring: + dashboards: + grafana: + # Auto-provisioning + provisioning: + enabled: true + folder: ContextFrame + + # Pre-built dashboards + dashboards: + - name: mcp-overview + uid: contextframe-mcp-overview + + - name: mcp-performance + uid: contextframe-mcp-performance + + - name: mcp-security + uid: contextframe-mcp-security +``` + +### Key Metrics to Display + +1. **Overview Dashboard** + - Request rate + - Error rate + - Response time (p50, p95, p99) + - Active users + - Document count + +2. **Performance Dashboard** + - Tool execution times + - Cache hit rates + - Queue depths + - Resource utilization + - Slow query log + +3. **Security Dashboard** + - Authentication attempts + - Failed authentications + - Rate limit violations + - Suspicious activity + - Audit trail + +## Cost Monitoring + +### Usage Tracking + +```yaml +monitoring: + cost: + enabled: true + + # Track costs per operation + operations: + document_create: + cost: 0.001 # $ per operation + search_documents: + cost: 0.002 + vector_search: + cost: 0.005 + + # Track by dimensions + dimensions: + - user + - tenant + - tool + - collection + + # Billing integration + billing: + provider: stripe + export_interval: daily +``` + +## Monitoring Best Practices + +### Development + +```yaml +# config/development.yaml +monitoring: + logging: + level: DEBUG + format: text + console: + enabled: true + colorize: true + metrics: + enabled: true + endpoint: + auth: false +``` + +### Production + +```yaml +# config/production.yaml +monitoring: + logging: + level: INFO + format: json + aggregation: + elasticsearch: + enabled: true + metrics: + enabled: true + backend: prometheus + tracing: + enabled: true + sampling: + rate: 0.1 + alerting: + enabled: true +``` + +### Monitoring Checklist + +- [ ] Configure structured logging +- [ ] Set appropriate log levels +- [ ] Enable metrics collection +- [ ] Set up distributed tracing +- [ ] Configure health checks +- [ ] Define alert rules +- [ ] Create dashboards +- [ ] Set up log aggregation +- [ ] Configure cost tracking +- [ ] Test alert delivery +- [ ] Document runbooks +- [ ] Regular review cycles + +## Troubleshooting + +### Missing Metrics + +```bash +# Check metrics endpoint +curl http://localhost:9090/metrics | grep mcp_ + +# Verify Prometheus scraping +curl http://prometheus:9090/api/v1/targets + +# Check metric labels +curl http://localhost:9090/metrics | grep -A5 "HELP mcp_" +``` + +### Log Issues + +```bash +# Check log output +tail -f /var/log/contextframe/mcp.log + +# Verify log format +jq . /var/log/contextframe/mcp.log + +# Test log aggregation +echo '{"test": "message"}' | nc fluentd 24224 +``` + +### Trace Problems + +```bash +# Check trace export +curl http://jaeger:16686/api/traces?service=mcp-server + +# Verify sampling +grep "trace_id" /var/log/contextframe/mcp.log + +# Test trace propagation +curl -H "traceparent: 00-abcd1234abcd1234abcd1234abcd1234-1234abcd1234abcd-01" \ + http://localhost:8000/health +``` + +## Next Steps + +- [Production Deployment](../guides/production-deployment.md) - Production monitoring setup +- [Performance Tuning](../guides/performance.md) - Optimize based on metrics +- [Troubleshooting Guide](../guides/troubleshooting.md) - Common issues +- [Runbook Templates](../reference/runbooks.md) - Operational procedures \ No newline at end of file diff --git a/docs/mcp/configuration/security.md b/docs/mcp/configuration/security.md new file mode 100644 index 0000000..243ac91 --- /dev/null +++ b/docs/mcp/configuration/security.md @@ -0,0 +1,762 @@ +# Security Configuration Guide + +This comprehensive guide covers all security aspects of the ContextFrame MCP server, from basic authentication to advanced authorization and audit logging. + +## Security Overview + +The MCP server implements defense-in-depth with multiple security layers: + +```mermaid +graph TD + A[Client Request] --> B[TLS/SSL] + B --> C[Authentication] + C --> D[Authorization] + D --> E[Rate Limiting] + E --> F[Input Validation] + F --> G[Audit Logging] + G --> H[Tool Execution] +``` + +## Authentication + +### API Key Authentication + +The simplest authentication method: + +```yaml +security: + auth: + type: api_key + api_key: + header: X-API-Key # Header name + keys: + - ${API_KEY_1} # Environment variable + - ${API_KEY_2} + hash_keys: true # Store hashed keys +``` + +Usage: +```bash +curl -H "X-API-Key: your-api-key" \ + http://localhost:8000/mcp/v1/tools/search_documents +``` + +### Bearer Token (JWT) + +For stateless authentication: + +```yaml +security: + auth: + type: bearer + jwt: + secret: ${JWT_SECRET} + algorithm: HS256 # HS256, RS256, ES256 + issuer: contextframe + audience: mcp-server + expiry: 3600 # seconds + + # For RS256/ES256 + public_key_file: /path/to/public.pem + private_key_file: /path/to/private.pem +``` + +Usage: +```bash +curl -H "Authorization: Bearer eyJhbGc..." \ + http://localhost:8000/mcp/v1/tools/search_documents +``` + +### OAuth 2.0 / OpenID Connect + +For enterprise SSO integration: + +```yaml +security: + auth: + type: oauth2 + oauth2: + provider: generic # generic, auth0, okta, azure-ad + issuer: https://auth.example.com + client_id: ${OAUTH_CLIENT_ID} + client_secret: ${OAUTH_CLIENT_SECRET} + redirect_uri: http://localhost:8000/auth/callback + scopes: + - openid + - profile + - email + + # Provider-specific + azure_ad: + tenant_id: ${AZURE_TENANT_ID} + + # Token validation + validate_issuer: true + validate_audience: true + required_claims: + - email_verified +``` + +### Multiple Authentication Methods + +Support multiple auth methods simultaneously: + +```yaml +security: + auth: + type: multiple + methods: + # API keys for service accounts + - type: api_key + header: X-Service-Key + required_for: + - /mcp/v1/admin/* + + # JWT for users + - type: bearer + jwt: + secret: ${JWT_SECRET} + + # OAuth for web apps + - type: oauth2 + oauth2: + provider: auth0 + + # Fallback order + priority: + - bearer + - oauth2 + - api_key +``` + +### Custom Authentication + +Implement custom authentication: + +```yaml +security: + auth: + type: custom + custom: + module: contextframe.auth.custom + class: LDAPAuthProvider + config: + server: ldap://corp.example.com + base_dn: dc=example,dc=com +``` + +## Authorization + +### Role-Based Access Control (RBAC) + +Define roles and permissions: + +```yaml +security: + authorization: + enabled: true + type: rbac + + # Define roles + roles: + # Read-only access + reader: + description: "Read-only access to documents" + permissions: + - documents:read + - collections:read + - search:* + + # Write access + writer: + description: "Read and write documents" + inherits: reader + permissions: + - documents:write + - collections:write + + # Admin access + admin: + description: "Full administrative access" + permissions: + - "*" # All permissions + + # User-role mapping + users: + alice@example.com: + roles: [writer] + bob@example.com: + roles: [reader] + + # Group-role mapping + groups: + engineering: + roles: [writer] + contractors: + roles: [reader] + + # Default role for new users + default_role: reader +``` + +### Resource-Based Access Control + +Fine-grained permissions per resource: + +```yaml +security: + authorization: + type: resource + + # Resource policies + policies: + # Collection-level access + - resource: collection:research-* + effect: allow + principals: + - role:researcher + actions: + - read + - write + + # Document-level access + - resource: document:* + effect: allow + principals: + - user:admin@example.com + actions: + - "*" + conditions: + - type: metadata + field: classification + operator: not_equals + value: confidential +``` + +### Tool-Based Permissions + +Control access to specific tools: + +```yaml +security: + authorization: + tools: + # Public tools + public: + - health_check + - list_tools + + # Authenticated tools + authenticated: + - search_documents + - document_get + - collection_list + + # Restricted tools + restricted: + admin: + - dataset_stats + - usage_metrics + - server_info + writer: + - document_create + - document_update + - document_delete +``` + +### Dynamic Authorization + +Implement dynamic authorization rules: + +```yaml +security: + authorization: + dynamic: + provider: opa # Open Policy Agent + endpoint: http://opa:8181 + + # Or custom provider + custom: + module: contextframe.auth.dynamic + class: DynamicAuthProvider + cache_ttl: 300 # seconds +``` + +## Rate Limiting + +Prevent abuse and ensure fair usage: + +```yaml +security: + rate_limiting: + enabled: true + + # Global limits + global: + requests: 1000 + window: 60 # seconds + + # Per-user limits + user: + requests: 100 + window: 60 + + # Per-IP limits + ip: + requests: 50 + window: 60 + + # Tool-specific limits + tools: + search_documents: + requests: 50 + window: 60 + cost: 2 # Cost units + + document_create: + requests: 20 + window: 60 + cost: 5 + + import_documents: + requests: 5 + window: 3600 # 1 hour + cost: 100 + + # Quota management + quotas: + enabled: true + default_quota: 10000 # Monthly units + + # Burst handling + burst: + enabled: true + multiplier: 2 # Allow 2x burst + + # Response headers + headers: + enabled: true + prefix: X-RateLimit- +``` + +### Rate Limit Responses + +```json +// Rate limit exceeded +{ + "error": { + "code": "RATE_LIMIT_EXCEEDED", + "message": "Too many requests", + "details": { + "limit": 100, + "window": 60, + "retry_after": 45, + "quota_remaining": 500 + } + } +} +``` + +## Audit Logging + +Track all security-relevant events: + +```yaml +security: + audit: + enabled: true + + # What to log + events: + - authentication.success + - authentication.failure + - authorization.denied + - document.created + - document.deleted + - collection.modified + - admin.action + + # Log detail level + detail_level: full # minimal, standard, full + + # Include request/response + include_payload: true + sanitize_sensitive: true + + # Storage backend + storage: + type: file # file, database, siem + file: + path: /var/log/contextframe/audit.log + rotation: daily + retention: 90 # days + format: json + + # Or database + database: + connection: postgresql://... + table: audit_logs + + # Or SIEM + siem: + type: splunk # splunk, elastic, datadog + endpoint: https://splunk.example.com + token: ${SPLUNK_TOKEN} +``` + +### Audit Log Format + +```json +{ + "timestamp": "2024-01-15T10:30:00Z", + "event_type": "document.created", + "user": { + "id": "user_123", + "email": "alice@example.com", + "ip": "192.168.1.100", + "user_agent": "MCP Client/1.0" + }, + "action": { + "tool": "document_create", + "parameters": { + "collection_id": "research" + } + }, + "result": { + "success": true, + "document_id": "doc_456" + }, + "metadata": { + "request_id": "req_789", + "session_id": "sess_012" + } +} +``` + +## Input Validation + +Protect against malicious input: + +```yaml +security: + validation: + # Size limits + limits: + max_document_size: 10485760 # 10MB + max_query_length: 1000 + max_metadata_size: 65536 # 64KB + max_batch_size: 100 + + # Content validation + content: + # Sanitize HTML/Scripts + sanitize_html: true + allowed_tags: [] + + # File type restrictions + allowed_content_types: + - text/plain + - text/markdown + - application/json + + # Metadata validation + metadata: + max_keys: 50 + max_key_length: 100 + max_value_length: 1000 + + # Reserved keys + reserved_keys: + - _id + - _created + - _updated + + # Query validation + query: + max_depth: 5 # Nested query depth + disable_regex: false # Disable regex queries + allowed_operators: + - equals + - contains + - greater_than + - less_than +``` + +## Transport Security + +### TLS/SSL Configuration + +```yaml +security: + tls: + enabled: true + + # Certificates + cert_file: /etc/ssl/certs/server.crt + key_file: /etc/ssl/private/server.key + ca_file: /etc/ssl/certs/ca-bundle.crt + + # Protocol settings + min_version: "1.2" # Minimum TLS version + ciphers: + - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 + - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 + + # Client certificates + client_auth: + enabled: false + required: false + ca_file: /etc/ssl/certs/client-ca.crt + + # HSTS + hsts: + enabled: true + max_age: 31536000 # 1 year + include_subdomains: true + preload: true +``` + +### CORS Configuration + +```yaml +security: + cors: + enabled: true + + # Allowed origins + origins: + - https://app.example.com + - https://*.example.com + + # Or allow credentials with specific origins + allow_credentials: true + origins: + - https://trusted-app.com + + # Methods and headers + methods: ["GET", "POST", "OPTIONS"] + headers: ["Content-Type", "Authorization", "X-API-Key"] + expose_headers: ["X-Request-ID", "X-RateLimit-*"] + max_age: 3600 +``` + +## Secrets Management + +### Environment Variables + +```yaml +security: + secrets: + # Reference environment variables + api_key: ${API_KEY} + jwt_secret: ${JWT_SECRET} + + # With defaults + session_secret: ${SESSION_SECRET:-generated-default} +``` + +### External Secret Stores + +```yaml +security: + secrets: + provider: vault # vault, aws-secrets, azure-keyvault + + # HashiCorp Vault + vault: + address: https://vault.example.com + token: ${VAULT_TOKEN} + path: secret/contextframe/mcp + + # AWS Secrets Manager + aws_secrets: + region: us-east-1 + prefix: /contextframe/mcp/ + + # Azure Key Vault + azure_keyvault: + vault_name: contextframe-kv + tenant_id: ${AZURE_TENANT_ID} +``` + +### Secret Rotation + +```yaml +security: + secrets: + rotation: + enabled: true + interval: 2592000 # 30 days + + # Graceful rotation + overlap_period: 86400 # 24 hours + + # Notification + notify: + webhook: https://ops.example.com/secret-rotated +``` + +## Security Headers + +Add security headers to responses: + +```yaml +security: + headers: + # Security headers + X-Content-Type-Options: nosniff + X-Frame-Options: DENY + X-XSS-Protection: "1; mode=block" + Referrer-Policy: strict-origin-when-cross-origin + + # CSP + Content-Security-Policy: "default-src 'self'; script-src 'self'" + + # Custom headers + custom: + X-Service-Version: ${VERSION} +``` + +## Security Monitoring + +### Security Metrics + +```yaml +monitoring: + security: + # Track security events + metrics: + - auth_attempts_total + - auth_failures_total + - rate_limit_hits_total + - invalid_token_total + + # Alerts + alerts: + - name: high_auth_failures + condition: rate(auth_failures_total[5m]) > 10 + severity: warning + + - name: suspected_attack + condition: rate(rate_limit_hits_total[1m]) > 100 + severity: critical +``` + +### Security Scanning + +```yaml +security: + scanning: + # Dependency scanning + dependencies: + enabled: true + schedule: daily + + # Container scanning + containers: + enabled: true + registry: registry.example.com + + # SAST/DAST + code_analysis: + enabled: true + tools: ["semgrep", "bandit"] +``` + +## Best Practices + +### Development + +```yaml +# config/development.yaml +security: + enabled: true # Always test with security + auth: + type: api_key + api_key: + keys: ["dev-key-123"] + audit: + enabled: true + detail_level: full +``` + +### Production + +```yaml +# config/production.yaml +security: + enabled: true + auth: + type: oauth2 + oauth2: + provider: auth0 + tls: + enabled: true + min_version: "1.2" + rate_limiting: + enabled: true + audit: + enabled: true + storage: + type: siem +``` + +### Security Checklist + +- [ ] Enable authentication +- [ ] Configure authorization rules +- [ ] Set up rate limiting +- [ ] Enable audit logging +- [ ] Configure TLS/SSL +- [ ] Implement input validation +- [ ] Set security headers +- [ ] Configure CORS properly +- [ ] Manage secrets securely +- [ ] Monitor security metrics +- [ ] Regular security updates +- [ ] Incident response plan + +## Troubleshooting + +### Authentication Issues + +```bash +# Test authentication +curl -v -H "Authorization: Bearer $TOKEN" \ + http://localhost:8000/mcp/v1/tools/health_check + +# Check JWT token +jwt decode $TOKEN + +# Validate API key +contextframe-mcp validate-auth --key $API_KEY +``` + +### Authorization Issues + +```bash +# Check user permissions +contextframe-mcp show-permissions --user alice@example.com + +# Test authorization +contextframe-mcp test-auth --user alice@example.com --tool document_create +``` + +### Rate Limiting Issues + +```bash +# Check rate limit status +curl -I http://localhost:8000/mcp/v1/tools/search_documents + +# Response headers +X-RateLimit-Limit: 100 +X-RateLimit-Remaining: 45 +X-RateLimit-Reset: 1705320000 +``` + +## Next Steps + +- [Monitoring Setup](monitoring.md) - Configure logging and metrics +- [Production Deployment](../guides/production-deployment.md) - Security hardening +- [Compliance Guide](../guides/compliance.md) - Meet regulatory requirements +- [Incident Response](../guides/incident-response.md) - Security procedures \ No newline at end of file diff --git a/docs/mcp/cookbook/index.md b/docs/mcp/cookbook/index.md new file mode 100644 index 0000000..8f8d7fc --- /dev/null +++ b/docs/mcp/cookbook/index.md @@ -0,0 +1,558 @@ +# MCP Cookbook + +Real-world examples and patterns for using the ContextFrame MCP server effectively. Each recipe demonstrates a complete, working solution to common use cases. + +## Quick Examples + +### Basic Document Search + +```python +from contextframe.mcp import MCPClient + +# Initialize client +client = MCPClient("http://localhost:8000") + +# Simple search +results = client.search_documents( + query="machine learning", + limit=10 +) + +# Print results +for doc in results['documents']: + print(f"Score: {doc['score']:.2f} - {doc['metadata'].get('title', 'Untitled')}") +``` + +### Create and Organize Documents + +```python +# Create a collection +collection = client.collection_create( + name="research-papers", + description="ML research papers collection" +) + +# Add documents to collection +doc_id = client.document_create( + content="# Attention Is All You Need\n\nThe Transformer architecture...", + metadata={ + "title": "Attention Is All You Need", + "authors": ["Vaswani et al."], + "year": 2017, + "tags": ["transformers", "attention", "nlp"] + }, + collection_id=collection['id'] +) + +print(f"Document {doc_id} added to collection {collection['id']}") +``` + +## Recipe Categories + +### 🔍 Search Patterns + +#### Semantic Search with Filtering + +```python +# Combine semantic search with metadata filters +results = client.search_documents( + query="explain transformer architecture", + search_type="hybrid", + metadata_filter={ + "year": {"$gte": 2020}, + "tags": {"$contains": "nlp"} + }, + limit=5 +) +``` + +#### Multi-Collection Search + +```python +# Search across specific collections +collections = ["research", "documentation", "tutorials"] +all_results = [] + +for coll_id in collections: + results = client.search_within_collection( + collection_id=coll_id, + query="deployment best practices", + limit=3 + ) + all_results.extend(results['documents']) + +# Sort by relevance score +all_results.sort(key=lambda x: x['score'], reverse=True) +``` + +### 📝 Document Management + +#### Bulk Import with Progress + +```python +import json +from pathlib import Path + +def import_documents_with_progress(directory_path): + """Import all JSON documents from a directory""" + documents = [] + + # Prepare documents + for file_path in Path(directory_path).glob("*.json"): + with open(file_path) as f: + data = json.load(f) + documents.append({ + "content": data['content'], + "metadata": { + **data.get('metadata', {}), + "source_file": file_path.name + } + }) + + # Batch import + result = client.document_create_batch( + documents=documents, + batch_size=50 + ) + + print(f"Imported {result['created']} documents") + print(f"Failed: {result['failed']}") + + return result['document_ids'] +``` + +#### Document Versioning + +```python +def create_document_version(doc_id, new_content, version_note): + """Create a new version of a document""" + + # Get current document + current = client.document_get(doc_id) + + # Create new version with updated content + new_version = client.document_create( + content=new_content, + metadata={ + **current['metadata'], + "previous_version": doc_id, + "version_note": version_note, + "version_date": datetime.utcnow().isoformat() + } + ) + + # Update original to point to new version + client.document_update( + id=doc_id, + metadata={ + **current['metadata'], + "latest_version": new_version['id'] + } + ) + + return new_version['id'] +``` + +### 🤖 AI Agent Patterns + +#### RAG with Source Citations + +```python +def rag_with_citations(question, llm_client): + """Answer questions with source citations""" + + # Search for relevant context + search_results = client.search_documents( + query=question, + limit=5, + search_type="hybrid" + ) + + # Build context with citations + context_parts = [] + citations = [] + + for i, doc in enumerate(search_results['documents']): + ref_num = i + 1 + context_parts.append(f"[{ref_num}] {doc['content'][:500]}...") + citations.append({ + "ref": ref_num, + "title": doc['metadata'].get('title', 'Untitled'), + "id": doc['id'] + }) + + # Generate answer with LLM + prompt = f"""Answer this question using the provided context. +Include citation numbers [n] when referencing sources. + +Context: +{chr(10).join(context_parts)} + +Question: {question} + +Answer:""" + + answer = llm_client.generate(prompt) + + return { + "answer": answer, + "citations": citations + } +``` + +#### Interactive Document Assistant + +```python +class DocumentAssistant: + def __init__(self, mcp_client, llm_client): + self.mcp = mcp_client + self.llm = llm_client + self.context_collection = None + + def set_context(self, collection_name): + """Focus on a specific collection""" + collections = self.mcp.collection_list() + for coll in collections['collections']: + if coll['name'] == collection_name: + self.context_collection = coll['id'] + return f"Context set to: {collection_name}" + return f"Collection '{collection_name}' not found" + + def ask(self, question): + """Answer questions about the documents""" + search_params = { + "query": question, + "limit": 5 + } + + if self.context_collection: + search_params["collection_id"] = self.context_collection + + results = self.mcp.search_documents(**search_params) + + if not results['documents']: + return "I couldn't find any relevant documents." + + # Use LLM to synthesize answer + context = "\n---\n".join([ + doc['content'][:1000] for doc in results['documents'] + ]) + + response = self.llm.generate( + f"Based on these documents, {question}\n\nDocuments:\n{context}" + ) + + return response + + def summarize_collection(self): + """Provide collection overview""" + if not self.context_collection: + return "No collection selected" + + stats = self.mcp.collection_stats(self.context_collection) + recent = self.mcp.collection_list_documents( + collection_id=self.context_collection, + limit=5, + sort_by="updated_at" + ) + + summary = f"""Collection Overview: +- Total documents: {stats['document_count']} +- Total size: {stats['total_size_bytes'] / 1024 / 1024:.1f} MB +- Created: {stats['created_at']} + +Recent documents:""" + + for doc in recent['documents']: + summary += f"\n- {doc['metadata'].get('title', doc['id'])}" + + return summary +``` + +### 📊 Analytics Patterns + +#### Usage Analytics Dashboard + +```python +def generate_usage_report(time_range="day"): + """Generate usage analytics report""" + + # Get usage metrics + metrics = client.usage_metrics(time_range=time_range) + + # Get popular searches + search_analytics = client.search_analytics( + time_range=time_range, + limit=10 + ) + + # Get dataset growth + stats = client.dataset_stats() + + report = f"""## Usage Report - {time_range} + +### Activity Summary +- Total API calls: {metrics['metrics']['total_requests']:,} +- Unique users: {metrics['metrics']['unique_users']} +- Average response time: {metrics['metrics']['avg_response_time_ms']}ms +- Error rate: {metrics['metrics']['error_rate']:.2%} + +### Popular Tools +""" + + for tool, count in metrics['metrics']['by_tool'].items(): + report += f"- {tool}: {count:,} calls\n" + + report += f"\n### Top Search Queries\n" + for query in search_analytics['queries']: + report += f"- \"{query['query']}\" ({query['count']} searches)\n" + + report += f"\n### Dataset Growth +- Total documents: {stats['total_documents']:,} +- Storage used: {stats['storage_size_bytes'] / 1024 / 1024 / 1024:.2f} GB +- Collections: {stats['total_collections']} +""" + + return report +``` + +#### Cost Attribution + +```python +def calculate_operation_costs(user_id, date_range): + """Calculate costs for a specific user""" + + # Define cost model + costs = { + "search_documents": 0.002, + "document_create": 0.001, + "document_update": 0.001, + "vector_search": 0.005, + "import_documents": 0.01 + } + + # Get user's usage + usage = client.usage_metrics( + time_range="custom", + start_date=date_range[0], + end_date=date_range[1], + user_id=user_id + ) + + total_cost = 0 + breakdown = {} + + for tool, count in usage['by_tool'].items(): + cost = costs.get(tool, 0.001) * count + total_cost += cost + breakdown[tool] = { + "count": count, + "unit_cost": costs.get(tool, 0.001), + "total": cost + } + + return { + "user_id": user_id, + "period": date_range, + "total_cost": round(total_cost, 2), + "breakdown": breakdown + } +``` + +### 🔒 Security Patterns + +#### Secure Document Sharing + +```python +def create_shared_collection(documents, allowed_users, expiry_days=7): + """Create a time-limited shared collection""" + + # Create collection with expiry + expiry_date = (datetime.utcnow() + timedelta(days=expiry_days)).isoformat() + + collection = client.collection_create( + name=f"shared_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}", + metadata={ + "type": "shared", + "expires_at": expiry_date, + "allowed_users": allowed_users, + "created_by": "current_user" + } + ) + + # Add documents to collection + doc_ids = [] + for doc in documents: + doc_id = client.document_create( + content=doc['content'], + metadata={ + **doc.get('metadata', {}), + "shared": True, + "expires_at": expiry_date + }, + collection_id=collection['id'] + ) + doc_ids.append(doc_id) + + # Generate share token (implement in your auth system) + share_token = generate_share_token( + collection_id=collection['id'], + allowed_users=allowed_users, + expiry=expiry_date + ) + + return { + "collection_id": collection['id'], + "share_token": share_token, + "expires_at": expiry_date, + "document_count": len(doc_ids) + } +``` + +### 🚀 Performance Patterns + +#### Parallel Processing + +```python +import asyncio +from contextframe.mcp.async_client import AsyncMCPClient + +async def parallel_document_processing(file_paths): + """Process multiple documents in parallel""" + + async with AsyncMCPClient("http://localhost:8000") as client: + tasks = [] + + for path in file_paths: + task = process_single_document(client, path) + tasks.append(task) + + # Process in parallel with concurrency limit + sem = asyncio.Semaphore(10) # Max 10 concurrent + + async def bounded_process(task): + async with sem: + return await task + + results = await asyncio.gather( + *[bounded_process(task) for task in tasks], + return_exceptions=True + ) + + # Handle results + successful = [r for r in results if not isinstance(r, Exception)] + failed = [r for r in results if isinstance(r, Exception)] + + print(f"Processed {len(successful)} documents successfully") + print(f"Failed: {len(failed)}") + + return successful + +async def process_single_document(client, file_path): + """Process a single document""" + with open(file_path) as f: + content = f.read() + + # Extract metadata (example) + metadata = { + "source": file_path, + "processed_at": datetime.utcnow().isoformat() + } + + # Create document + result = await client.document_create( + content=content, + metadata=metadata + ) + + return result +``` + +## Testing Patterns + +### Integration Testing + +```python +import pytest +from contextframe.mcp import MCPClient + +@pytest.fixture +def mcp_test_client(): + """Test client with isolated dataset""" + client = MCPClient("http://localhost:8000/test") + yield client + # Cleanup + client.clear_dataset() + +def test_search_functionality(mcp_test_client): + """Test search returns relevant results""" + # Create test documents + docs = [ + {"content": "Python programming guide", "metadata": {"type": "guide"}}, + {"content": "JavaScript tutorial", "metadata": {"type": "tutorial"}}, + {"content": "Python data science", "metadata": {"type": "guide"}} + ] + + doc_ids = [] + for doc in docs: + result = mcp_test_client.document_create(**doc) + doc_ids.append(result['id']) + + # Test search + results = mcp_test_client.search_documents( + query="Python", + limit=10 + ) + + assert results['total_count'] == 2 + assert all("Python" in doc['content'] for doc in results['documents']) + + # Cleanup + for doc_id in doc_ids: + mcp_test_client.document_delete(doc_id) +``` + +## Error Handling Patterns + +### Resilient Client + +```python +class ResilientMCPClient: + def __init__(self, base_url, max_retries=3): + self.client = MCPClient(base_url) + self.max_retries = max_retries + + def execute_with_retry(self, func, *args, **kwargs): + """Execute function with exponential backoff""" + last_error = None + + for attempt in range(self.max_retries): + try: + return func(*args, **kwargs) + except RateLimitError as e: + # Wait for rate limit reset + time.sleep(e.retry_after) + except NetworkError as e: + # Exponential backoff + wait_time = 2 ** attempt + time.sleep(wait_time) + last_error = e + except Exception as e: + # Don't retry on other errors + raise + + raise last_error + + def search_documents(self, **kwargs): + return self.execute_with_retry( + self.client.search_documents, + **kwargs + ) +``` + +## Next Steps + +- [API Reference](../api/tools.md) - Complete tool documentation +- [Integration Guide](../guides/agent-integration.md) - Connect your AI agents +- [Security Guide](../configuration/security.md) - Secure your deployment +- [Performance Guide](../guides/performance.md) - Optimize for scale \ No newline at end of file diff --git a/docs/mcp/getting-started/installation.md b/docs/mcp/getting-started/installation.md new file mode 100644 index 0000000..a1542dd --- /dev/null +++ b/docs/mcp/getting-started/installation.md @@ -0,0 +1,329 @@ +# Installation Guide + +This guide covers all installation options for the ContextFrame MCP server, from quick setup to production deployment. + +## System Requirements + +### Minimum Requirements + +- **Python**: 3.10, 3.11, or 3.12 +- **Memory**: 2GB RAM +- **Storage**: 1GB free space (plus dataset storage) +- **OS**: Linux, macOS, or Windows (WSL2 recommended) + +### Recommended for Production + +- **Python**: 3.11+ (better performance) +- **Memory**: 8GB+ RAM +- **Storage**: SSD with sufficient space for datasets +- **OS**: Linux (Ubuntu 22.04 LTS or similar) + +## Installation Methods + +### 1. PyPI Installation (Recommended) + +The simplest way to install ContextFrame with MCP server: + +```bash +# Basic installation +pip install contextframe[mcp] + +# With all optional dependencies +pip install contextframe[mcp,monitoring,auth] + +# For development +pip install contextframe[mcp,dev] +``` + +Verify installation: +```bash +contextframe-mcp --version +# contextframe-mcp 0.1.0 +``` + +### 2. Docker Installation + +For containerized deployments: + +```bash +# Pull the latest image +docker pull contextframe/mcp-server:latest + +# Run with default settings +docker run -p 8000:8000 contextframe/mcp-server + +# Run with custom configuration +docker run -p 8000:8000 \ + -v /path/to/config:/config \ + -v /path/to/data:/data \ + -e CONTEXTFRAME_CONFIG=/config/mcp.yaml \ + contextframe/mcp-server +``` + +Docker Compose example: +```yaml +version: '3.8' + +services: + mcp-server: + image: contextframe/mcp-server:latest + ports: + - "8000:8000" + volumes: + - ./data:/data + - ./config:/config + environment: + - CONTEXTFRAME_DATASET_PATH=/data/dataset.lance + - CONTEXTFRAME_API_KEY=${API_KEY} + restart: unless-stopped +``` + +### 3. From Source + +For development or customization: + +```bash +# Clone the repository +git clone https://github.com/greyhaven-ai/contextframe.git +cd contextframe + +# Create virtual environment +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate + +# Install in development mode +pip install -e ".[mcp,dev]" + +# Run tests to verify +pytest tests/mcp/ +``` + +### 4. Using UV (Fast Python Package Manager) + +For faster installation with UV: + +```bash +# Install UV if not already installed +curl -LsSf https://astral.sh/uv/install.sh | sh + +# Install ContextFrame with UV +uv pip install contextframe[mcp] + +# Or in development mode +uv pip install -e ".[mcp,dev]" +``` + +## Platform-Specific Instructions + +### Linux (Ubuntu/Debian) + +```bash +# Install system dependencies +sudo apt-get update +sudo apt-get install -y python3-pip python3-venv + +# Install ContextFrame +pip install contextframe[mcp] +``` + +### macOS + +```bash +# Using Homebrew +brew install python@3.11 + +# Install ContextFrame +pip3 install contextframe[mcp] +``` + +### Windows + +#### Option 1: WSL2 (Recommended) + +```bash +# In WSL2 Ubuntu +sudo apt-get update +sudo apt-get install -y python3-pip + +pip install contextframe[mcp] +``` + +#### Option 2: Native Windows + +```powershell +# Install Python from python.org +# Then in PowerShell: +pip install contextframe[mcp] +``` + +## Post-Installation Setup + +### 1. Initialize a Dataset + +Create your first dataset: + +```bash +# Create a new dataset +contextframe-mcp init-dataset ~/contextframe-data/my-dataset.lance + +# Verify dataset +contextframe-mcp validate-dataset ~/contextframe-data/my-dataset.lance +``` + +### 2. Create Configuration + +Generate a default configuration: + +```bash +# Generate config file +contextframe-mcp generate-config > contextframe-mcp.yaml + +# Edit as needed +nano contextframe-mcp.yaml +``` + +### 3. Test the Installation + +Start the server and run health check: + +```bash +# Start server +contextframe-mcp serve --config contextframe-mcp.yaml + +# In another terminal, check health +curl http://localhost:8000/health +``` + +## Dependency Management + +### Core Dependencies + +- `lance`: Columnar storage format +- `fastapi`: HTTP server framework +- `pydantic`: Data validation +- `numpy`: Numerical operations +- `typer`: CLI framework + +### Optional Dependencies + +Install based on your needs: + +```bash +# For monitoring +pip install contextframe[monitoring] # Adds: prometheus-client, opentelemetry + +# For advanced auth +pip install contextframe[auth] # Adds: python-jose, passlib + +# For cloud storage +pip install contextframe[cloud] # Adds: boto3, google-cloud-storage, azure-storage + +# All extras +pip install contextframe[all] +``` + +## Upgrading + +### From PyPI + +```bash +# Upgrade to latest version +pip install --upgrade contextframe[mcp] + +# Upgrade to specific version +pip install contextframe[mcp]==0.2.0 +``` + +### Docker + +```bash +# Pull latest image +docker pull contextframe/mcp-server:latest + +# Or specific version +docker pull contextframe/mcp-server:0.2.0 +``` + +## Uninstallation + +### Remove PyPI Installation + +```bash +# Uninstall package +pip uninstall contextframe + +# Remove configuration files (optional) +rm -rf ~/.contextframe/ +``` + +### Remove Docker + +```bash +# Stop and remove container +docker stop contextframe-mcp +docker rm contextframe-mcp + +# Remove image +docker rmi contextframe/mcp-server:latest +``` + +## Troubleshooting Installation + +### Common Issues + +#### Python Version Mismatch + +```bash +# Check Python version +python --version + +# Use specific Python version +python3.11 -m pip install contextframe[mcp] +``` + +#### Permission Errors + +```bash +# Install for current user only +pip install --user contextframe[mcp] + +# Or use virtual environment (recommended) +python -m venv venv +source venv/bin/activate +pip install contextframe[mcp] +``` + +#### Missing System Dependencies + +```bash +# On Ubuntu/Debian +sudo apt-get install python3-dev build-essential + +# On macOS +xcode-select --install +``` + +### Verification Steps + +1. Check installation: +```bash +pip show contextframe +contextframe-mcp --help +``` + +2. Test import: +```python +python -c "import contextframe.mcp; print('Success!')" +``` + +3. Run built-in tests: +```bash +contextframe-mcp test-installation +``` + +## Next Steps + +- [Quick Start Guide](quickstart.md) - Get running in 5 minutes +- [Configuration Guide](../configuration/index.md) - Configure for your needs +- [Security Setup](../configuration/security.md) - Enable authentication +- [Production Deployment](../guides/production-deployment.md) - Scale for production \ No newline at end of file diff --git a/docs/mcp/getting-started/quickstart.md b/docs/mcp/getting-started/quickstart.md new file mode 100644 index 0000000..2b6615a --- /dev/null +++ b/docs/mcp/getting-started/quickstart.md @@ -0,0 +1,311 @@ +# Quick Start Guide + +Get the ContextFrame MCP server running in 5 minutes! This guide will walk you through installation, basic configuration, and your first MCP operations. + +## Prerequisites + +- Python 3.10+ or Docker +- Basic understanding of REST APIs +- A ContextFrame dataset (we'll create one if you don't have one) + +## Installation + +Choose your preferred installation method: + +### Option 1: Using pip + +```bash +# Install the ContextFrame package with MCP server +pip install contextframe[mcp] + +# Verify installation +contextframe-mcp --version +``` + +### Option 2: Using Docker + +```bash +# Pull the MCP server image +docker pull contextframe/mcp-server:latest + +# Run the server +docker run -p 8000:8000 contextframe/mcp-server +``` + +### Option 3: From source + +```bash +# Clone the repository +git clone https://github.com/greyhaven-ai/contextframe.git +cd contextframe + +# Install in development mode +pip install -e ".[mcp,dev]" +``` + +## Starting the Server + +### Basic HTTP Server + +Start the MCP server with default settings: + +```bash +# Start on default port 8000 +contextframe-mcp serve + +# Or specify a custom port +contextframe-mcp serve --port 3000 + +# Enable development mode with auto-reload +contextframe-mcp serve --dev +``` + +You should see: +``` +INFO: Started ContextFrame MCP server on http://0.0.0.0:8000 +INFO: API documentation at http://0.0.0.0:8000/docs +INFO: MCP endpoint at http://0.0.0.0:8000/mcp/v1 +``` + +### Stdio Mode (for local integrations) + +For direct integration with AI agents that support stdio: + +```bash +# Start in stdio mode +contextframe-mcp stdio + +# Or with a specific dataset +contextframe-mcp stdio --dataset-path /path/to/dataset.lance +``` + +## Your First MCP Operation + +### 1. Check Server Health + +Verify the server is running: + +```bash +curl http://localhost:8000/health +``` + +Response: +```json +{ + "status": "healthy", + "version": "0.1.0", + "transport": "http", + "tools_available": 43 +} +``` + +### 2. List Available Tools + +See what operations are available: + +```bash +curl http://localhost:8000/mcp/v1/tools +``` + +This returns all 43 available tools with their descriptions and parameters. + +### 3. Create Your First Document + +Let's create a document using the MCP protocol: + +```bash +curl -X POST http://localhost:8000/mcp/v1/tools/document_create \ + -H "Content-Type: application/json" \ + -d '{ + "params": { + "content": "# My First Document\n\nThis is my first document in ContextFrame!", + "metadata": { + "title": "My First Document", + "type": "markdown", + "tags": ["quickstart", "demo"] + } + } + }' +``` + +Response: +```json +{ + "result": { + "id": "doc_abc123", + "dataset_id": "default", + "message": "Document created successfully" + } +} +``` + +### 4. Search for Documents + +Now let's search for our document: + +```bash +curl -X POST http://localhost:8000/mcp/v1/tools/search_documents \ + -H "Content-Type: application/json" \ + -d '{ + "params": { + "query": "first document", + "limit": 5 + } + }' +``` + +## Using with AI Agents + +### Claude Desktop + +Add to your Claude Desktop configuration: + +```json +{ + "mcpServers": { + "contextframe": { + "command": "contextframe-mcp", + "args": ["stdio"], + "env": { + "DATASET_PATH": "/path/to/your/dataset.lance" + } + } + } +} +``` + +### Python Client + +```python +from contextframe.mcp import MCPClient + +# Connect to the MCP server +client = MCPClient("http://localhost:8000") + +# Create a document +doc_id = client.document_create( + content="# Project Notes\n\nImportant project information...", + metadata={"project": "alpha", "type": "notes"} +) + +# Search documents +results = client.search_documents( + query="project information", + limit=10 +) + +# Get collection stats +stats = client.collection_stats(collection_id="default") +print(f"Documents: {stats['document_count']}") +print(f"Total size: {stats['total_size_bytes']:,} bytes") +``` + +### Using with curl + +For quick testing and scripting: + +```bash +# Create a collection +curl -X POST http://localhost:8000/mcp/v1/tools/collection_create \ + -H "Content-Type: application/json" \ + -d '{ + "params": { + "name": "research-papers", + "description": "Academic research papers" + } + }' + +# List collections +curl -X POST http://localhost:8000/mcp/v1/tools/collection_list \ + -H "Content-Type: application/json" \ + -d '{"params": {}}' +``` + +## Configuration Basics + +### Environment Variables + +```bash +# Set the default dataset path +export CONTEXTFRAME_DATASET_PATH=/data/my-dataset.lance + +# Enable development mode +export CONTEXTFRAME_DEV_MODE=true + +# Set custom host and port +export CONTEXTFRAME_HOST=0.0.0.0 +export CONTEXTFRAME_PORT=3000 + +# Enable API key authentication +export CONTEXTFRAME_API_KEY=your-secret-key +``` + +### Configuration File + +Create a `contextframe-mcp.yaml`: + +```yaml +server: + host: 0.0.0.0 + port: 8000 + +dataset: + path: /data/my-dataset.lance + +security: + enabled: true + api_key: ${CONTEXTFRAME_API_KEY} + +monitoring: + enabled: true + level: INFO +``` + +## What's Next? + +Now that you have the MCP server running: + +1. **[Explore Core Concepts](../concepts/overview.md)** - Understand how MCP works +2. **[Review API Reference](../api/tools.md)** - See all available tools +3. **[Configure Security](../configuration/security.md)** - Set up authentication +4. **[Try Examples](../cookbook/index.md)** - Real-world usage patterns +5. **[Integrate with AI](../guides/agent-integration.md)** - Connect your agents + +## Troubleshooting + +### Server won't start + +```bash +# Check if port is already in use +lsof -i :8000 + +# Try a different port +contextframe-mcp serve --port 3001 +``` + +### Can't connect to server + +```bash +# Verify server is running +curl http://localhost:8000/health + +# Check firewall settings +# Ensure the port is open +``` + +### Dataset not found + +```bash +# Create a new dataset +contextframe-mcp init-dataset /path/to/new/dataset.lance + +# Or specify existing dataset +contextframe-mcp serve --dataset-path /path/to/existing.lance +``` + +## Getting Help + +- Check the [FAQ](../reference/faq.md) +- Review [error codes](../reference/errors.md) +- Ask in [GitHub Discussions](https://github.com/greyhaven-ai/contextframe/discussions) +- Report issues on [GitHub](https://github.com/greyhaven-ai/contextframe/issues) \ No newline at end of file diff --git a/docs/mcp/guides/agent-integration.md b/docs/mcp/guides/agent-integration.md new file mode 100644 index 0000000..9eb5098 --- /dev/null +++ b/docs/mcp/guides/agent-integration.md @@ -0,0 +1,600 @@ +# AI Agent Integration Guide + +This guide covers how to integrate various AI agents and language models with the ContextFrame MCP server. Whether you're using Claude, GPT, or custom agents, this guide will help you connect and configure your integration. + +## Integration Overview + +```mermaid +graph LR + subgraph "AI Agents" + A[Claude Desktop] + B[OpenAI GPTs] + C[LangChain] + D[Custom Agents] + end + + subgraph "MCP Server" + E[HTTP API] + F[Stdio Interface] + end + + subgraph "ContextFrame" + G[Document Store] + end + + A --> F + B --> E + C --> E + D --> E + E --> G + F --> G +``` + +## Claude Desktop Integration + +### Configuration + +Add to your Claude Desktop configuration file: + +**macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` +**Windows**: `%APPDATA%\Claude\claude_desktop_config.json` + +```json +{ + "mcpServers": { + "contextframe": { + "command": "contextframe-mcp", + "args": ["stdio"], + "env": { + "CONTEXTFRAME_DATASET_PATH": "/path/to/your/dataset.lance", + "CONTEXTFRAME_LOG_LEVEL": "INFO" + } + } + } +} +``` + +### Advanced Configuration + +```json +{ + "mcpServers": { + "contextframe": { + "command": "contextframe-mcp", + "args": [ + "stdio", + "--config", "/path/to/mcp-config.yaml" + ], + "env": { + "CONTEXTFRAME_API_KEY": "your-api-key", + "CONTEXTFRAME_SECURITY_ENABLED": "true" + }, + "metadata": { + "description": "Company knowledge base", + "capabilities": ["search", "document-management"] + } + } + } +} +``` + +### Usage in Claude + +Once configured, you can use natural language commands: + +``` +"Search our documentation for deployment procedures" +"Create a new document with the meeting notes I just shared" +"Find all documents related to the Q4 project" +"Show me the latest updates to our API documentation" +``` + +## OpenAI GPT Integration + +### Custom GPT Action + +Create a custom GPT with the following action schema: + +```yaml +openapi: 3.0.0 +info: + title: ContextFrame MCP API + version: 1.0.0 +servers: + - url: https://your-mcp-server.com +paths: + /mcp/v1/tools/search_documents: + post: + operationId: searchDocuments + summary: Search documents + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + params: + type: object + properties: + query: + type: string + limit: + type: integer + default: 10 + responses: + '200': + description: Search results + content: + application/json: + schema: + type: object +``` + +### Authentication Setup + +Configure OAuth or API Key authentication: + +```json +{ + "authentication": { + "type": "api_key", + "api_key": { + "header": "X-API-Key", + "value": "{{SECRET_API_KEY}}" + } + } +} +``` + +## LangChain Integration + +### Python Integration + +```python +from langchain.tools import Tool +from contextframe.mcp import MCPClient +import json + +# Initialize MCP client +mcp_client = MCPClient( + base_url="http://localhost:8000", + api_key="your-api-key" +) + +# Create LangChain tools +def search_documents(query: str) -> str: + """Search ContextFrame documents""" + results = mcp_client.search_documents( + query=query, + limit=5 + ) + return json.dumps(results, indent=2) + +def create_document(content: str, metadata: dict) -> str: + """Create a new document in ContextFrame""" + doc_id = mcp_client.document_create( + content=content, + metadata=metadata + ) + return f"Document created with ID: {doc_id}" + +# Register tools +search_tool = Tool( + name="search_contextframe", + func=search_documents, + description="Search documents in ContextFrame" +) + +create_tool = Tool( + name="create_document", + func=create_document, + description="Create a new document in ContextFrame" +) + +# Use with an agent +from langchain.agents import initialize_agent, AgentType +from langchain.llms import OpenAI + +llm = OpenAI() +tools = [search_tool, create_tool] + +agent = initialize_agent( + tools=tools, + llm=llm, + agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, + verbose=True +) + +# Example usage +result = agent.run("Search for documents about machine learning") +``` + +### Async Integration + +```python +import asyncio +from langchain.tools import StructuredTool +from contextframe.mcp.async_client import AsyncMCPClient + +async def async_search(query: str, limit: int = 10) -> dict: + """Async document search""" + async with AsyncMCPClient("http://localhost:8000") as client: + return await client.search_documents( + query=query, + limit=limit + ) + +# Create async tool +async_search_tool = StructuredTool.from_function( + coroutine=async_search, + name="async_search", + description="Asynchronously search documents" +) +``` + +## Custom Agent Integration + +### HTTP API Client + +Basic implementation for any language: + +```python +import requests +import json + +class ContextFrameMCP: + def __init__(self, base_url, api_key=None): + self.base_url = base_url + self.headers = { + "Content-Type": "application/json" + } + if api_key: + self.headers["X-API-Key"] = api_key + + def call_tool(self, tool_name, params): + """Call any MCP tool""" + url = f"{self.base_url}/mcp/v1/tools/{tool_name}" + response = requests.post( + url, + headers=self.headers, + json={"params": params} + ) + response.raise_for_status() + return response.json() + + def search(self, query, **kwargs): + """Search documents""" + return self.call_tool("search_documents", { + "query": query, + **kwargs + }) + + def create_document(self, content, metadata=None): + """Create a document""" + return self.call_tool("document_create", { + "content": content, + "metadata": metadata or {} + }) + +# Usage +client = ContextFrameMCP("http://localhost:8000", "your-api-key") +results = client.search("AI research", limit=5) +``` + +### WebSocket Streaming + +For real-time updates: + +```javascript +class MCPStreamClient { + constructor(url, apiKey) { + this.url = url; + this.apiKey = apiKey; + } + + streamSearch(query, onResult) { + const ws = new WebSocket(`${this.url}/mcp/v1/stream`); + + ws.onopen = () => { + ws.send(JSON.stringify({ + tool: "search_documents", + params: { query, stream: true }, + auth: { api_key: this.apiKey } + })); + }; + + ws.onmessage = (event) => { + const result = JSON.parse(event.data); + onResult(result); + }; + + return ws; + } +} + +// Usage +const client = new MCPStreamClient("ws://localhost:8000", "api-key"); +const ws = client.streamSearch("quantum computing", (result) => { + console.log("Found:", result); +}); +``` + +## Integration Patterns + +### 1. RAG (Retrieval Augmented Generation) + +```python +class RAGAgent: + def __init__(self, mcp_client, llm): + self.mcp = mcp_client + self.llm = llm + + def answer_question(self, question): + # 1. Search for relevant documents + context_docs = self.mcp.search_documents( + query=question, + limit=5, + search_type="hybrid" + ) + + # 2. Build context + context = "\n\n".join([ + f"Document {i+1}: {doc['content'][:500]}..." + for i, doc in enumerate(context_docs['documents']) + ]) + + # 3. Generate answer + prompt = f"""Based on the following context, answer the question. + +Context: +{context} + +Question: {question} + +Answer:""" + + return self.llm.generate(prompt) +``` + +### 2. Document Processing Pipeline + +```python +class DocumentProcessor: + def __init__(self, mcp_client): + self.mcp = mcp_client + + async def process_documents(self, documents): + # 1. Create collection for batch + collection = await self.mcp.collection_create({ + "name": f"batch_{datetime.now().isoformat()}", + "description": "Processing batch" + }) + + # 2. Process and store documents + for doc in documents: + # Extract metadata + metadata = self.extract_metadata(doc) + + # Generate embeddings (if not using auto-embed) + embeddings = self.generate_embeddings(doc['content']) + + # Store document + doc_id = await self.mcp.document_create({ + "content": doc['content'], + "metadata": metadata, + "embeddings": embeddings, + "collection_id": collection['id'] + }) + + # Track progress + yield {"processed": doc_id, "status": "success"} +``` + +### 3. Interactive Assistant + +```python +class InteractiveAssistant: + def __init__(self, mcp_client): + self.mcp = mcp_client + self.conversation_history = [] + + def handle_command(self, command): + # Parse intent + if "search for" in command: + query = command.replace("search for", "").strip() + return self.search_and_summarize(query) + + elif "create document about" in command: + topic = command.replace("create document about", "").strip() + return self.create_document_interactive(topic) + + elif "analyze" in command: + return self.analyze_collection() + + def search_and_summarize(self, query): + results = self.mcp.search_documents(query=query, limit=3) + + if not results['documents']: + return "No documents found." + + summary = f"Found {results['total_count']} documents:\n" + for doc in results['documents']: + summary += f"- {doc['metadata'].get('title', 'Untitled')}\n" + + return summary +``` + +## Best Practices + +### 1. Authentication & Security + +Always use proper authentication in production: + +```python +# Good: Use environment variables +mcp_client = MCPClient( + base_url=os.getenv("MCP_SERVER_URL"), + api_key=os.getenv("MCP_API_KEY") +) + +# Good: Use secure token storage +from keyring import get_password +api_key = get_password("contextframe", "api_key") + +# Bad: Hardcoded credentials +mcp_client = MCPClient( + base_url="http://localhost:8000", + api_key="hardcoded-key-123" # Never do this! +) +``` + +### 2. Error Handling + +Implement robust error handling: + +```python +def safe_search(mcp_client, query): + try: + results = mcp_client.search_documents(query=query) + return results + except RateLimitError as e: + # Handle rate limiting + time.sleep(e.retry_after) + return safe_search(mcp_client, query) + except AuthenticationError: + # Re-authenticate + mcp_client.refresh_token() + return safe_search(mcp_client, query) + except MCPError as e: + # Log and handle gracefully + logger.error(f"MCP Error: {e}") + return {"documents": [], "error": str(e)} +``` + +### 3. Performance Optimization + +```python +# Use batch operations +documents = [...] # List of documents +mcp_client.document_create_batch(documents) # Single request + +# Cache frequently accessed data +from functools import lru_cache + +@lru_cache(maxsize=100) +def get_collection_info(collection_id): + return mcp_client.collection_get(collection_id) + +# Use streaming for large results +for batch in mcp_client.search_documents_stream(query, batch_size=50): + process_batch(batch) +``` + +### 4. Context Management + +Manage context window efficiently: + +```python +def optimize_context(documents, max_tokens=4000): + """Select most relevant document chunks for context""" + chunks = [] + current_tokens = 0 + + for doc in documents: + # Estimate tokens (rough approximation) + tokens = len(doc['content']) // 4 + + if current_tokens + tokens > max_tokens: + # Truncate or skip + remaining = max_tokens - current_tokens + truncated = doc['content'][:remaining * 4] + chunks.append(truncated) + break + else: + chunks.append(doc['content']) + current_tokens += tokens + + return "\n\n".join(chunks) +``` + +## Monitoring Integration + +### Metrics Collection + +```python +from prometheus_client import Counter, Histogram + +# Define metrics +search_counter = Counter('mcp_searches_total', 'Total MCP searches') +search_duration = Histogram('mcp_search_duration_seconds', 'Search duration') + +# Instrument your code +@search_duration.time() +def monitored_search(query): + search_counter.inc() + return mcp_client.search_documents(query=query) +``` + +### Logging Integration + +```python +import logging +import json + +# Configure structured logging +logger = logging.getLogger("mcp_integration") + +def log_mcp_operation(operation, params, result): + logger.info(json.dumps({ + "operation": operation, + "params": params, + "result_count": len(result.get('documents', [])), + "execution_time": result.get('metadata', {}).get('execution_time_ms'), + "timestamp": datetime.utcnow().isoformat() + })) +``` + +## Troubleshooting + +### Connection Issues + +```bash +# Test connectivity +curl http://your-mcp-server:8000/health + +# Test authentication +curl -H "X-API-Key: your-key" \ + http://your-mcp-server:8000/mcp/v1/tools/list_tools +``` + +### Common Errors + +1. **Rate Limiting** + ```python + # Check headers for rate limit info + response.headers['X-RateLimit-Remaining'] + response.headers['X-RateLimit-Reset'] + ``` + +2. **Timeout Issues** + ```python + # Increase timeout for large operations + mcp_client = MCPClient( + base_url="...", + timeout=60 # 60 seconds + ) + ``` + +3. **Memory Issues** + ```python + # Use streaming for large datasets + # Don't load all results at once + ``` + +## Next Steps + +- [Production Deployment](production-deployment.md) - Deploy your integration +- [Performance Tuning](performance.md) - Optimize for scale +- [Security Best Practices](../configuration/security.md) - Secure your integration +- [Cookbook Examples](../cookbook/index.md) - More integration patterns \ No newline at end of file diff --git a/docs/mcp/guides/cli-tools.md b/docs/mcp/guides/cli-tools.md new file mode 100644 index 0000000..69c797e --- /dev/null +++ b/docs/mcp/guides/cli-tools.md @@ -0,0 +1,259 @@ +# CLI Tools and Bash Scripts + +ContextFrame provides two ways to interact via command line: the MCP server for AI agents and direct bash scripts for human users and simple integrations. + +## Overview + +```mermaid +graph LR + subgraph "Users" + A[Human Users] + B[Shell Scripts] + C[AI Agents] + end + + subgraph "Interfaces" + D[Bash Scripts] + E[MCP Server] + end + + subgraph "ContextFrame" + F[Lance Dataset] + end + + A --> D + B --> D + C --> E + D --> F + E --> F +``` + +## Bash Scripts (Direct CLI) + +ContextFrame includes bash script wrappers in `contextframe/scripts/` that provide direct command-line access to core functionality. These are ideal for: + +- Human users working in the terminal +- Shell script automation +- Simple integrations +- Quick operations without running a server + +### Available Scripts + +1. **contextframe-search** - Search documents using text, vector, or hybrid search +2. **contextframe-add** - Add documents to a dataset with optional chunking and embeddings +3. **contextframe-get** - Retrieve specific documents by ID +4. **contextframe-list** - List and filter documents in a dataset + +### Quick Examples + +```bash +# Search for documents +contextframe-search data.lance "machine learning" --limit 10 + +# Add a document +contextframe-add data.lance document.txt --collection "docs" --embeddings + +# Get a specific document +contextframe-get data.lance doc-123 --format markdown + +# List documents in a collection +contextframe-list data.lance --collection "research" --format json +``` + +### Installation + +Add to your PATH: +```bash +export PATH="$PATH:/path/to/contextframe/contextframe/scripts" +``` + +Or create symlinks: +```bash +ln -s /path/to/contextframe/contextframe/scripts/contextframe-* /usr/local/bin/ +``` + +### When to Use Bash Scripts + +- **Quick Operations**: One-off searches or document additions +- **Shell Integration**: Part of bash pipelines or scripts +- **Human Interface**: Direct terminal usage +- **Simple Automation**: Cron jobs, basic workflows + +## MCP Server (AI Agent Interface) + +The MCP server provides a standardized protocol for AI agents with advanced features: + +### Features Beyond Bash Scripts + +1. **Stateful Operations**: Maintain context across requests +2. **Batch Processing**: Efficient bulk operations +3. **Real-time Updates**: Streaming and subscriptions +4. **Advanced Security**: Authentication, authorization, rate limiting +5. **Monitoring**: Metrics, logging, tracing +6. **43+ Tools**: Comprehensive tool suite + +### Quick Start + +```bash +# Start MCP server +contextframe-mcp serve --port 8000 + +# Or in stdio mode for Claude Desktop +contextframe-mcp stdio +``` + +### When to Use MCP Server + +- **AI Agent Integration**: Claude, GPT, LangChain +- **Production Systems**: Scalable, monitored deployments +- **Complex Operations**: Transactions, streaming, subscriptions +- **Multi-user Access**: Authentication and authorization +- **API Access**: RESTful HTTP interface + +## Comparison + +| Feature | Bash Scripts | MCP Server | +|---------|-------------|------------| +| **Interface** | Command line | HTTP/stdio API | +| **Target Users** | Humans, shell scripts | AI agents, applications | +| **Setup** | Just PATH config | Run server process | +| **Authentication** | File system permissions | API keys, OAuth | +| **Operations** | Basic CRUD + search | 43+ advanced tools | +| **Performance** | Single operations | Batch, streaming | +| **Monitoring** | Basic exit codes | Full metrics/logging | +| **State** | Stateless | Stateful sessions | + +## Migration Path + +### From Bash Scripts to MCP + +If you're currently using bash scripts and want to migrate to MCP: + +```python +# Bash script approach +subprocess.run(["contextframe-search", "data.lance", "query"]) + +# MCP client approach +from contextframe.mcp import MCPClient +client = MCPClient("http://localhost:8000") +results = client.search_documents(query="query") +``` + +### Using Both Together + +You can use both interfaces in the same workflow: + +```bash +# Use bash script for one-off import +contextframe-add data.lance documents/*.txt --embeddings + +# Then use MCP server for application +contextframe-mcp serve --dataset data.lance +``` + +## Environment Configuration + +Both interfaces share embedding configuration: + +```bash +# OpenAI embeddings +export OPENAI_API_KEY="sk-..." +export CONTEXTFRAME_EMBED_MODEL="text-embedding-ada-002" + +# Works for both: +contextframe-add data.lance doc.txt --embeddings # Bash script +# AND +client.document_create(content="...", generate_embeddings=True) # MCP +``` + +## Examples + +### Bash Script Pipeline + +```bash +#!/bin/bash +# Simple document processing pipeline + +# Add documents +find ./docs -name "*.md" | while read file; do + contextframe-add knowledge.lance "$file" \ + --collection "documentation" \ + --chunk-size 1000 +done + +# Search and process results +contextframe-search knowledge.lance "$1" --format json | \ + jq '.results[] | {id, score, title: .metadata.title}' +``` + +### MCP Integration + +```python +# AI agent with context +class ContextAgent: + def __init__(self): + self.mcp = MCPClient("http://localhost:8000") + + def answer_with_context(self, question): + # Get relevant context + results = self.mcp.search_documents( + query=question, + limit=5 + ) + + # Use context in response + context = "\n".join([doc['content'] for doc in results['documents']]) + return self.generate_answer(question, context) +``` + +## Best Practices + +### For Bash Scripts + +1. **Use for Simple Tasks**: One-off operations, quick searches +2. **Pipe-Friendly**: Design scripts to work in pipelines +3. **Error Handling**: Check exit codes +4. **JSON Output**: Use `--format json` for parsing + +### For MCP Server + +1. **Use for Applications**: AI agents, web apps, services +2. **Monitor Performance**: Enable metrics and logging +3. **Secure Access**: Always use authentication in production +4. **Batch Operations**: Group related operations + +## Troubleshooting + +### Bash Scripts + +```bash +# Debug mode +export DEBUG=1 +contextframe-search data.lance "test" + +# Check script location +which contextframe-search + +# Verify dataset +ls -la data.lance/ +``` + +### MCP Server + +```bash +# Check server health +curl http://localhost:8000/health + +# View logs +contextframe-mcp serve --log-level DEBUG + +# Test with curl +curl -X POST http://localhost:8000/mcp/v1/tools/list_tools +``` + +## Next Steps + +- **Bash Scripts**: See [full documentation](https://github.com/contextframe/contextframe/blob/main/contextframe/scripts/README.md) +- **MCP Quick Start**: Follow the [5-minute guide](../getting-started/quickstart.md) +- **AI Integration**: See [agent integration guide](agent-integration.md) +- **Production Setup**: Check [deployment guide](production-deployment.md) \ No newline at end of file diff --git a/docs/mcp/index.md b/docs/mcp/index.md new file mode 100644 index 0000000..d3022fb --- /dev/null +++ b/docs/mcp/index.md @@ -0,0 +1,141 @@ +# ContextFrame MCP Server + +The ContextFrame Model Context Protocol (MCP) server provides a standardized interface for AI agents and language models to interact with ContextFrame datasets. Built on the MCP specification, it enables seamless integration with any MCP-compatible client. + +## What is MCP? + +The Model Context Protocol (MCP) is an open standard that enables seamless communication between AI models and external systems. ContextFrame's MCP server implements this protocol to provide: + +- **Standardized Access**: Uniform interface for querying and managing document collections +- **Tool-based Interaction**: Rich set of tools for document operations, search, and analysis +- **Transport Flexibility**: Support for HTTP (primary) and stdio transports +- **Production Ready**: Built-in security, monitoring, and performance optimization + +## Key Features + +### 🔧 Comprehensive Tool Suite + +- **Document Operations**: Create, read, update, delete documents +- **Advanced Search**: Full-text, semantic, and hybrid search capabilities +- **Collection Management**: Organize documents into logical collections +- **Batch Processing**: Efficient bulk operations with transactional support +- **Analytics**: Usage tracking, performance monitoring, and cost attribution + +### 🔒 Enterprise Security + +- **Multi-Provider Authentication**: API keys, OAuth 2.1, JWT tokens +- **Fine-grained Authorization**: Role-based and resource-level access control +- **Rate Limiting**: Protect against abuse with configurable limits +- **Comprehensive Audit Logging**: Track all operations for compliance + +### 📊 Built-in Monitoring + +- **Performance Metrics**: Track operation latency and throughput +- **Usage Analytics**: Monitor document and query patterns +- **Cost Attribution**: Calculate and track LLM operation costs +- **Real-time Dashboards**: Visualize system health and usage + +### 🚀 Production Ready + +- **HTTP-First Design**: RESTful API with optional SSE streaming +- **Horizontal Scalability**: Stateless design for easy scaling +- **Error Handling**: Graceful degradation and detailed error messages +- **Extensive Testing**: Comprehensive test coverage + +## Quick Navigation + +
+ +- :material-rocket-launch: **[Getting Started](getting-started/quickstart.md)** + + Get up and running with the MCP server in 5 minutes + +- :material-book-open: **[Core Concepts](concepts/overview.md)** + + Understand the fundamental concepts and architecture + +- :material-tools: **[API Reference](api/tools.md)** + + Complete reference for all MCP tools and operations + +- :material-console: **[CLI Tools](guides/cli-tools.md)** + + Direct command-line access with bash scripts + +- :material-shield-check: **[Security Guide](configuration/security.md)** + + Configure authentication, authorization, and security features + +- :material-chart-line: **[Monitoring Setup](configuration/monitoring.md)** + + Set up metrics collection and performance monitoring + +- :material-puzzle: **[Integration Guides](guides/agent-integration.md)** + + Connect your AI agents and applications to ContextFrame + +
+ +## Use Cases + +The ContextFrame MCP server is ideal for: + +- **AI Agent Development**: Provide agents with structured access to document collections +- **RAG Applications**: Build retrieval-augmented generation systems with rich context +- **Document Intelligence**: Extract insights from large document repositories +- **Knowledge Management**: Organize and query organizational knowledge bases +- **Research Tools**: Analyze and synthesize information from diverse sources + +## Architecture Overview + +```mermaid +graph TB + subgraph "AI Clients" + A[AI Agents] + B[LLM Applications] + C[MCP Clients] + end + + subgraph "MCP Server" + D[HTTP/stdio Transport] + E[Security Layer] + F[Tool Registry] + G[Monitoring] + end + + subgraph "ContextFrame" + H[Lance Dataset] + I[Document Store] + J[Vector Index] + end + + A --> D + B --> D + C --> D + D --> E + E --> F + F --> H + E --> G + G --> H + H --> I + H --> J +``` + +## Next Steps + +1. **[Install and Configure](getting-started/installation.md)** - Set up your MCP server +2. **[Follow the Quickstart](getting-started/quickstart.md)** - Build your first integration +3. **[Explore Tools](api/tools.md)** - Learn about available operations +4. **[Configure Security](configuration/security.md)** - Set up authentication and authorization +5. **[Join the Community](https://github.com/greyhaven-ai/contextframe)** - Get help and contribute + +## Getting Help + +- **Documentation**: You're in the right place! +- **GitHub Issues**: [Report bugs or request features](https://github.com/greyhaven-ai/contextframe/issues) +- **Discussions**: [Ask questions and share ideas](https://github.com/greyhaven-ai/contextframe/discussions) +- **Examples**: Check out the [cookbook](cookbook/index.md) for practical examples + +## License + +ContextFrame is open source software licensed under the MIT License. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 667197d..af5ccd8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -133,6 +133,32 @@ nav: - Object Storage: 'integration/object_storage.md' - Blobs: 'integration/blobs.md' + - MCP Server: + - Overview: 'mcp/index.md' + - Getting Started: + - Installation: 'mcp/getting-started/installation.md' + - Quick Start: 'mcp/getting-started/quickstart.md' + - Core Concepts: + - Overview: 'mcp/concepts/overview.md' + - Tools: 'mcp/concepts/tools.md' + - Transport: 'mcp/concepts/transport.md' + - API Reference: + - Tools Reference: 'mcp/api/tools.md' + - Configuration: + - Overview: 'mcp/configuration/index.md' + - Security: 'mcp/configuration/security.md' + - Monitoring: 'mcp/configuration/monitoring.md' + - Guides: + - CLI Tools & Bash Scripts: 'mcp/guides/cli-tools.md' + - Agent Integration: 'mcp/guides/agent-integration.md' + - Production Deployment: 'mcp/guides/production-deployment.md' + - Performance Tuning: 'mcp/guides/performance.md' + - Cookbook: + - Examples: 'mcp/cookbook/index.md' + - Reference: + - Error Codes: 'mcp/reference/errors.md' + - FAQ: 'mcp/reference/faq.md' + - CLI Reference: - Community: