feat: Implement Phase 3.5 & 3.6 - HTTP Transport and Analytics (CFOS-27)

[jayscambler]

Summary

This PR implements Phase 3.5 (HTTP Transport), Phase 3.6 (Analytics & Performance), and HTTP transport prioritization (CFOS-43), completing all of Phase 3.

Phase 3.5: HTTP/SSE Transport

Key Features

• HTTP Transport Adapter: Full-featured HTTP transport with SSE support
• Authentication: Multiple auth methods (API key, Bearer token, Basic auth)
• Security Middleware: Rate limiting, CORS, request validation
• SSE Support: Real-time streaming with automatic reconnection
• Connection Management: Graceful shutdown and cleanup

Components

• HTTPTransportAdapter: Main transport implementation
• AuthMiddleware: Flexible authentication system
• SecurityMiddleware: Rate limiting and validation
• SSEFormatter: Event formatting for Server-Sent Events
• HTTPServer: FastAPI-based server with routing

Phase 3.6: Analytics & Performance

Analytics Classes

• StatsCollector: Comprehensive dataset statistics collection
• QueryAnalyzer: Query performance tracking and analysis
• StorageOptimizer: Dataset optimization operations
• IndexAdvisor: Intelligent index recommendations

MCP Tools (8 new tools)

1. get_dataset_stats: Dataset statistics and metrics
2. analyze_usage: Usage pattern analysis
3. query_performance: Query performance tracking
4. relationship_analysis: Relationship graph analytics
5. optimize_storage: Storage optimization operations
6. index_recommendations: Index creation suggestions
7. benchmark_operations: Performance benchmarking
8. export_metrics: Export analytics in multiple formats

FrameDataset Enhancements

Added Lance-native analytics methods:

• get_dataset_stats(): Comprehensive dataset statistics
• get_fragment_stats(): Fragment-level statistics
• compact_files(): File compaction for optimization
• optimize_indices(): Index optimization
• cleanup_old_versions(): Version cleanup
• list_indices(): List all dataset indices
• get_version_history(): Version history with metadata

HTTP Transport Prioritization (CFOS-43)

Changes Made

1. Set HTTP as default transport in MCPConfig (was stdio)
2. Updated documentation to clarify HTTP is primary, SSE is optional:
   • Added TRANSPORT_GUIDE.md with clear guidance
   • Created http_client_example.py demonstrating HTTP-first approach
   • Added http_primary_transport.md implementation notes
3. Updated code comments to clarify SSE is optional
4. Created comprehensive tests:
   • test_http_first_approach.py: Validates HTTP as primary
   • test_http_primary.py: Demonstrates recommended patterns
   • test_transport_migration.py: Migration scenarios

Key Clarifications

• HTTP with simple JSON responses is the primary transport
• SSE is optional and only for specific streaming scenarios:
  • Progress tracking: /mcp/v1/sse/progress/{id}
  • Subscriptions: /mcp/v1/sse/subscribe
• Main endpoint (/mcp/v1/jsonrpc) uses standard HTTP POST/JSON

Testing

✅ All tests passing:

• HTTP transport tests: Authentication, SSE, security features
• Analytics tests: 11 comprehensive tests covering all components
• HTTP-first tests: 14 tests validating transport priorities
• Full mock coverage for Lance operations

Phase 3 Complete

This PR completes all of Phase 3:

• ✅ Phase 3.1: Transport abstraction
• ✅ Phase 3.2: Batch operations
• ✅ Phase 3.3: Collection management
• ✅ Phase 3.4: Subscription system
• ✅ Phase 3.5: HTTP/SSE transport
• ✅ Phase 3.6: Analytics & performance
• ✅ CFOS-43: HTTP transport prioritization

Ready for Phase 4: Production Ready features!

Related to #27, #44

[linear]

CFOS-27 MCP Server Components