diff --git a/README.md b/README.md index 7b115e7..170a2aa 100644 --- a/README.md +++ b/README.md @@ -19,47 +19,76 @@ In contrast to other protocols, UTCP places a strong emphasis on: ![MCP vs. UTCP](https://github.com/user-attachments/assets/3cadfc19-8eea-4467-b606-66e580b89444) -## New Architecture in 1.0.0 +## Repository Structure -UTCP has been refactored into a core library and a set of optional plugins. +This repository contains the complete UTCP Python implementation: + +- **[`core/`](core/)** - Core `utcp` package with foundational components ([README](core/README.md)) +- **[`plugins/communication_protocols/`](plugins/communication_protocols/)** - Protocol-specific plugins: + - [`http/`](plugins/communication_protocols/http/) - HTTP/REST, SSE, streaming, OpenAPI ([README](plugins/communication_protocols/http/README.md)) + - [`cli/`](plugins/communication_protocols/cli/) - Command-line tools ([README](plugins/communication_protocols/cli/README.md)) + - [`mcp/`](plugins/communication_protocols/mcp/) - Model Context Protocol ([README](plugins/communication_protocols/mcp/README.md)) + - [`text/`](plugins/communication_protocols/text/) - File-based tools ([README](plugins/communication_protocols/text/README.md)) + - [`socket/`](plugins/communication_protocols/socket/) - TCP/UDP (🚧 In Progress) + - [`gql/`](plugins/communication_protocols/gql/) - GraphQL (🚧 In Progress) + +## Architecture Overview + +UTCP uses a modular architecture with a core library and protocol plugins: ### Core Package (`utcp`) -The `utcp` package provides the central components and interfaces: -* **Data Models**: Pydantic models for `Tool`, `CallTemplate`, `UtcpManual`, and `Auth`. -* **Pluggable Interfaces**: - * `CommunicationProtocol`: Defines the contract for protocol-specific communication (e.g., HTTP, CLI). - * `ConcurrentToolRepository`: An interface for storing and retrieving tools with thread-safe access. - * `ToolSearchStrategy`: An interface for implementing tool search algorithms. - * `VariableSubstitutor`: Handles variable substitution in configurations. - * `ToolPostProcessor`: Allows for modifying tool results before they are returned. -* **Default Implementations**: - * `UtcpClient`: The main client for interacting with the UTCP ecosystem. - * `InMemToolRepository`: An in-memory tool repository with asynchronous read-write locks. - * `TagAndDescriptionWordMatchStrategy`: An improved search strategy that matches on tags and description keywords. - -### Protocol Plugins - -Communication protocols are now separate, installable packages. This keeps the core lean and allows users to install only the protocols they need. -* `utcp-http`: Supports HTTP, SSE, and streamable HTTP, plus an OpenAPI converter. -* `utcp-cli`: For wrapping local command-line tools. -* `utcp-mcp`: For interoperability with the Model Context Protocol (MCP). -* `utcp-text`: For reading text files. -* `utcp-socket`: Scaffolding for TCP and UDP protocols. (Work in progress, requires update) -* `utcp-gql`: Scaffolding for GraphQL. (Work in progress, requires update) - -## Installation - -Install the core library and any required protocol plugins. +The [`core/`](core/) directory contains the foundational components: +- **Data Models**: Pydantic models for `Tool`, `CallTemplate`, `UtcpManual`, and `Auth` +- **Client Interface**: Main `UtcpClient` for tool interaction +- **Plugin System**: Extensible interfaces for protocols, repositories, and search +- **Default Implementations**: Built-in tool storage and search strategies + +## Quick Start + +### Installation + +Install the core library and any required protocol plugins: ```bash -# Install the core client and the HTTP plugin +# Install core + HTTP plugin (most common) pip install utcp utcp-http -# Install the CLI plugin as well -pip install utcp-cli +# Install additional plugins as needed +pip install utcp-cli utcp-mcp utcp-text ``` +### Basic Usage + +```python +from utcp.utcp_client import UtcpClient + +# Create client with HTTP API +client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "my_api", + "call_template_type": "http", + "url": "https://api.example.com/utcp" + }] +}) + +# Call a tool +result = await client.call_tool("my_api.get_data", {"id": "123"}) +``` + +## Protocol Plugins + +UTCP supports multiple communication protocols through dedicated plugins: + +| Plugin | Description | Status | Documentation | +|--------|-------------|--------|---------------| +| [`utcp-http`](plugins/communication_protocols/http/) | HTTP/REST APIs, SSE, streaming | ✅ Stable | [HTTP Plugin README](plugins/communication_protocols/http/README.md) | +| [`utcp-cli`](plugins/communication_protocols/cli/) | Command-line tools | ✅ Stable | [CLI Plugin README](plugins/communication_protocols/cli/README.md) | +| [`utcp-mcp`](plugins/communication_protocols/mcp/) | Model Context Protocol | ✅ Stable | [MCP Plugin README](plugins/communication_protocols/mcp/README.md) | +| [`utcp-text`](plugins/communication_protocols/text/) | Local file-based tools | ✅ Stable | [Text Plugin README](plugins/communication_protocols/text/README.md) | +| [`utcp-socket`](plugins/communication_protocols/socket/) | TCP/UDP protocols | 🚧 In Progress | [Socket Plugin README](plugins/communication_protocols/socket/README.md) | +| [`utcp-gql`](plugins/communication_protocols/gql/) | GraphQL APIs | 🚧 In Progress | [GraphQL Plugin README](plugins/communication_protocols/gql/README.md) | + For development, you can install the packages in editable mode from the cloned repository: ```bash diff --git a/core/README.md b/core/README.md deleted file mode 100644 index 3eee92e..0000000 --- a/core/README.md +++ /dev/null @@ -1,494 +0,0 @@ -# Universal Tool Calling Protocol (UTCP) 1.0.1 - -[![Follow Org](https://img.shields.io/github/followers/universal-tool-calling-protocol?label=Follow%20Org&logo=github)](https://github.com/universal-tool-calling-protocol) -[![PyPI Downloads](https://static.pepy.tech/badge/utcp)](https://pepy.tech/projects/utcp) -[![License](https://img.shields.io/github/license/universal-tool-calling-protocol/python-utcp)](https://github.com/universal-tool-calling-protocol/python-utcp/blob/main/LICENSE) -[![CDTM S23](https://img.shields.io/badge/CDTM-S23-0b84f3)](https://cdtm.com/) - -## Introduction - -The Universal Tool Calling Protocol (UTCP) is a modern, flexible, and scalable standard for defining and interacting with tools across a wide variety of communication protocols. UTCP 1.0.0 introduces a modular core with a plugin-based architecture, making it more extensible, testable, and easier to package. - -In contrast to other protocols, UTCP places a strong emphasis on: - -* **Scalability**: UTCP is designed to handle a large number of tools and providers without compromising performance. -* **Extensibility**: A pluggable architecture allows developers to easily add new communication protocols, tool storage mechanisms, and search strategies without modifying the core library. -* **Interoperability**: With a growing ecosystem of protocol plugins (including HTTP, SSE, CLI, and more), UTCP can integrate with almost any existing service or infrastructure. -* **Ease of Use**: The protocol is built on simple, well-defined Pydantic models, making it easy for developers to implement and use. - - -![MCP vs. UTCP](https://github.com/user-attachments/assets/3cadfc19-8eea-4467-b606-66e580b89444) - -## New Architecture in 1.0.0 - -UTCP has been refactored into a core library and a set of optional plugins. - -### Core Package (`utcp`) - -The `utcp` package provides the central components and interfaces: -* **Data Models**: Pydantic models for `Tool`, `CallTemplate`, `UtcpManual`, and `Auth`. -* **Pluggable Interfaces**: - * `CommunicationProtocol`: Defines the contract for protocol-specific communication (e.g., HTTP, CLI). - * `ConcurrentToolRepository`: An interface for storing and retrieving tools with thread-safe access. - * `ToolSearchStrategy`: An interface for implementing tool search algorithms. - * `VariableSubstitutor`: Handles variable substitution in configurations. - * `ToolPostProcessor`: Allows for modifying tool results before they are returned. -* **Default Implementations**: - * `UtcpClient`: The main client for interacting with the UTCP ecosystem. - * `InMemToolRepository`: An in-memory tool repository with asynchronous read-write locks. - * `TagAndDescriptionWordMatchStrategy`: An improved search strategy that matches on tags and description keywords. - -### Protocol Plugins - -Communication protocols are now separate, installable packages. This keeps the core lean and allows users to install only the protocols they need. -* `utcp-http`: Supports HTTP, SSE, and streamable HTTP, plus an OpenAPI converter. -* `utcp-cli`: For wrapping local command-line tools. -* `utcp-mcp`: For interoperability with the Model Context Protocol (MCP). -* `utcp-text`: For reading text files. -* `utcp-socket`: Scaffolding for TCP and UDP protocols. (Work in progress, requires update) -* `utcp-gql`: Scaffolding for GraphQL. (Work in progress, requires update) - -## Installation - -Install the core library and any required protocol plugins. - -```bash -# Install the core client and the HTTP plugin -pip install utcp utcp-http - -# Install the CLI plugin as well -pip install utcp-cli -``` - -For development, you can install the packages in editable mode from the cloned repository: - -```bash -# Clone the repository -git clone https://github.com/universal-tool-calling-protocol/python-utcp.git -cd python-utcp - -# Install the core package in editable mode with dev dependencies -pip install -e core[dev] - -# Install a specific protocol plugin in editable mode -pip install -e plugins/communication_protocols/http -``` - -## Migration Guide from 0.x to 1.0.0 - -Version 1.0.0 introduces several breaking changes. Follow these steps to migrate your project. - -1. **Update Dependencies**: Install the new `utcp` core package and the specific protocol plugins you use (e.g., `utcp-http`, `utcp-cli`). -2. **Configuration**: - * **Configuration Object**: `UtcpClient` is initialized with a `UtcpClientConfig` object, dict or a path to a JSON file containing the configuration. - * **Manual Call Templates**: The `providers_file_path` option is removed. Instead of a file path, you now provide a list of `manual_call_templates` directly within the `UtcpClientConfig`. - * **Terminology**: The term `provider` has been replaced with `call_template`, and `provider_type` is now `call_template_type`. - * **Streamable HTTP**: The `call_template_type` `http_stream` has been renamed to `streamable_http`. -3. **Update Imports**: Change your imports to reflect the new modular structure. For example, `from utcp.client.transport_interfaces.http_transport import HttpProvider` becomes `from utcp_http.http_call_template import HttpCallTemplate`. -4. **Tool Search**: If you were using the default search, the new strategy is `TagAndDescriptionWordMatchStrategy`. This is the new default and requires no changes unless you were implementing a custom strategy. -5. **Tool Naming**: Tool names are now namespaced as `manual_name.tool_name`. The client handles this automatically. -6 **Variable Substitution Namespacing**: Variables that are subsituted in different `call_templates`, are first namespaced with the name of the manual with the `_` duplicated. So a key in a tool call template called `API_KEY` from the manual `manual_1` would be converted to `manual__1_API_KEY`. - -## Usage Examples - -### 1. Using the UTCP Client - -**`config.json`** (Optional) - -You can define a comprehensive client configuration in a JSON file. All of these fields are optional. - -```json -{ - "variables": { - "openlibrary_URL": "https://openlibrary.org/static/openapi.json" - }, - "load_variables_from": [ - { - "variable_loader_type": "dotenv", - "env_file_path": ".env" - } - ], - "tool_repository": { - "tool_repository_type": "in_memory" - }, - "tool_search_strategy": { - "tool_search_strategy_type": "tag_and_description_word_match" - }, - "manual_call_templates": [ - { - "name": "openlibrary", - "call_template_type": "http", - "http_method": "GET", - "url": "${URL}", - "content_type": "application/json" - }, - ], - "post_processing": [ - { - "tool_post_processor_type": "filter_dict", - "only_include_keys": ["name", "key"], - "only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"] - } - ] -} -``` - -**`client.py`** - -```python -import asyncio -from utcp.utcp_client import UtcpClient -from utcp.data.utcp_client_config import UtcpClientConfig - -async def main(): - # The UtcpClient can be created with a config file path, a dict, or a UtcpClientConfig object. - - # Option 1: Initialize from a config file path - # client_from_file = await UtcpClient.create(config="./config.json") - - # Option 2: Initialize from a dictionary - client_from_dict = await UtcpClient.create(config={ - "variables": { - "openlibrary_URL": "https://openlibrary.org/static/openapi.json" - }, - "load_variables_from": [ - { - "variable_loader_type": "dotenv", - "env_file_path": ".env" - } - ], - "tool_repository": { - "tool_repository_type": "in_memory" - }, - "tool_search_strategy": { - "tool_search_strategy_type": "tag_and_description_word_match" - }, - "manual_call_templates": [ - { - "name": "openlibrary", - "call_template_type": "http", - "http_method": "GET", - "url": "${URL}", - "content_type": "application/json" - } - ], - "post_processing": [ - { - "tool_post_processor_type": "filter_dict", - "only_include_keys": ["name", "key"], - "only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"] - } - ] - }) - - # Option 3: Initialize with a full-featured UtcpClientConfig object - from utcp_http.http_call_template import HttpCallTemplate - from utcp.data.variable_loader import VariableLoaderSerializer - from utcp.interfaces.tool_post_processor import ToolPostProcessorConfigSerializer - - config_obj = UtcpClientConfig( - variables={"openlibrary_URL": "https://openlibrary.org/static/openapi.json"}, - load_variables_from=[ - VariableLoaderSerializer().validate_dict({ - "variable_loader_type": "dotenv", "env_file_path": ".env" - }) - ], - manual_call_templates=[ - HttpCallTemplate( - name="openlibrary", - call_template_type="http", - http_method="GET", - url="${URL}", - content_type="application/json" - ) - ], - post_processing=[ - ToolPostProcessorConfigSerializer().validate_dict({ - "tool_post_processor_type": "filter_dict", - "only_include_keys": ["name", "key"], - "only_include_tools": ["openlibrary.read_search_authors_json_search_authors_json_get"] - }) - ] - ) - client = await UtcpClient.create(config=config_obj) - - # Call a tool. The name is namespaced: `manual_name.tool_name` - result = await client.call_tool( - tool_name="openlibrary.read_search_authors_json_search_authors_json_get", - tool_args={"q": "J. K. Rowling"} - ) - - print(result) - -if __name__ == "__main__": - asyncio.run(main()) -``` - -### 2. Providing a UTCP Manual - -A `UTCPManual` describes the tools you offer. The key change is replacing `tool_provider` with `call_template`. - -**`server.py`** - -UTCP decorator version: - -```python -from fastapi import FastAPI -from utcp_http.http_call_template import HttpCallTemplate -from utcp.data.utcp_manual import UtcpManual -from utcp.python_specific_tooling.tool_decorator import utcp_tool - -app = FastAPI() - -# The discovery endpoint returns the tool manual -@app.get("/utcp") -def utcp_discovery(): - return UtcpManual.create_from_decorators(manual_version="1.0.0") - -# The actual tool endpoint -@utcp_tool(tool_call_template=HttpCallTemplate( - name="get_weather", - url=f"https://example.com/api/weather", - http_method="GET" -), tags=["weather"]) -@app.get("/api/weather") -def get_weather(location: str): - return {"temperature": 22.5, "conditions": "Sunny"} -``` - - -No UTCP dependencies server version: - -```python -from fastapi import FastAPI - -app = FastAPI() - -# The discovery endpoint returns the tool manual -@app.get("/utcp") -def utcp_discovery(): - return { - "manual_version": "1.0.0", - "utcp_version": "1.0.1", - "tools": [ - { - "name": "get_weather", - "description": "Get current weather for a location", - "tags": ["weather"], - "inputs": { - "type": "object", - "properties": { - "location": {"type": "string"} - } - }, - "outputs": { - "type": "object", - "properties": { - "temperature": {"type": "number"}, - "conditions": {"type": "string"} - } - }, - "call_template": { - "call_template_type": "http", - "url": "https://example.com/api/weather", - "http_method": "GET" - } - } - ] - } - -# The actual tool endpoint -@app.get("/api/weather") -def get_weather(location: str): - return {"temperature": 22.5, "conditions": "Sunny"} -``` - -### 3. Full examples - -You can find full examples in the [examples repository](https://github.com/universal-tool-calling-protocol/utcp-examples). - -## Protocol Specification - -### `UtcpManual` and `Tool` Models - -The `tool_provider` object inside a `Tool` has been replaced by `call_template`. - -```json -{ - "manual_version": "string", - "utcp_version": "string", - "tools": [ - { - "name": "string", - "description": "string", - "inputs": { ... }, - "outputs": { ... }, - "tags": ["string"], - "call_template": { - "call_template_type": "http", - "url": "https://...", - "http_method": "GET" - } - } - ] -} -``` - -## Call Template Configuration Examples - -Configuration examples for each protocol. Remember to replace `provider_type` with `call_template_type`. - -### HTTP Call Template - -```json -{ - "name": "my_rest_api", - "call_template_type": "http", // Required - "url": "https://api.example.com/users/{user_id}", // Required - "http_method": "POST", // Required, default: "GET" - "content_type": "application/json", // Optional, default: "application/json" - "auth": { // Optional, example using ApiKeyAuth for a Bearer token. The client must prepend "Bearer " to the token. - "auth_type": "api_key", - "api_key": "Bearer $API_KEY", // Required - "var_name": "Authorization", // Optional, default: "X-Api-Key" - "location": "header" // Optional, default: "header" - }, - "headers": { // Optional - "X-Custom-Header": "value" - }, - "body_field": "body", // Optional, default: "body" - "header_fields": ["user_id"] // Optional -} -``` - -### SSE (Server-Sent Events) Call Template - -```json -{ - "name": "my_sse_stream", - "call_template_type": "sse", // Required - "url": "https://api.example.com/events", // Required - "event_type": "message", // Optional - "reconnect": true, // Optional, default: true - "retry_timeout": 30000, // Optional, default: 30000 (ms) - "auth": { // Optional, example using BasicAuth - "auth_type": "basic", - "username": "${USERNAME}", // Required - "password": "${PASSWORD}" // Required - }, - "headers": { // Optional - "X-Client-ID": "12345" - }, - "body_field": null, // Optional - "header_fields": [] // Optional -} -``` - -### Streamable HTTP Call Template - -Note the name change from `http_stream` to `streamable_http`. - -```json -{ - "name": "streaming_data_source", - "call_template_type": "streamable_http", // Required - "url": "https://api.example.com/stream", // Required - "http_method": "POST", // Optional, default: "GET" - "content_type": "application/octet-stream", // Optional, default: "application/octet-stream" - "chunk_size": 4096, // Optional, default: 4096 - "timeout": 60000, // Optional, default: 60000 (ms) - "auth": null, // Optional - "headers": {}, // Optional - "body_field": "data", // Optional - "header_fields": [] // Optional -} -``` - -### CLI Call Template - -```json -{ - "name": "my_cli_tool", - "call_template_type": "cli", // Required - "command_name": "my-command --utcp", // Required - "env_vars": { // Optional - "MY_VAR": "my_value" - }, - "working_dir": "/path/to/working/directory", // Optional - "auth": null // Optional (always null for CLI) -} -``` - -### Text Call Template - -```json -{ - "name": "my_text_manual", - "call_template_type": "text", // Required - "file_path": "./manuals/my_manual.json", // Required - "auth": null // Optional (always null for Text) -} -``` - -### MCP (Model Context Protocol) Call Template - -```json -{ - "name": "my_mcp_server", - "call_template_type": "mcp", // Required - "config": { // Required - "mcpServers": { - "server_name": { - "transport": "stdio", - "command": ["python", "-m", "my_mcp_server"] - } - } - }, - "auth": { // Optional, example using OAuth2 - "auth_type": "oauth2", - "token_url": "https://auth.example.com/token", // Required - "client_id": "${CLIENT_ID}", // Required - "client_secret": "${CLIENT_SECRET}", // Required - "scope": "read:tools" // Optional - } -} -``` - -## Testing - -The testing structure has been updated to reflect the new core/plugin split. - -### Running Tests - -To run all tests for the core library and all plugins: -```bash -# Ensure you have installed all dev dependencies -python -m pytest -``` - -To run tests for a specific package (e.g., the core library): -```bash -python -m pytest core/tests/ -``` - -To run tests for a specific plugin (e.g., HTTP): -```bash -python -m pytest plugins/communication_protocols/http/tests/ -v -``` - -To run tests with coverage: -```bash -python -m pytest --cov=utcp --cov-report=xml -``` - -## Build - -The build process now involves building each package (`core` and `plugins`) separately if needed, though they are published to PyPI independently. - -1. Create and activate a virtual environment. -2. Install build dependencies: `pip install build`. -3. Navigate to the package directory (e.g., `cd core`). -4. Run the build: `python -m build`. -5. The distributable files (`.whl` and `.tar.gz`) will be in the `dist/` directory. - -## [Contributors](https://www.utcp.io/about) diff --git a/core/src/utcp/exceptions/utcp_serializer_validation_error.py b/core/src/utcp/exceptions/utcp_serializer_validation_error.py index 1a935df..98bafde 100644 --- a/core/src/utcp/exceptions/utcp_serializer_validation_error.py +++ b/core/src/utcp/exceptions/utcp_serializer_validation_error.py @@ -1,3 +1,12 @@ class UtcpSerializerValidationError(Exception): """REQUIRED - Exception raised when a serializer validation fails.""" + Exception raised when a serializer validation fails. + + Thrown by serializers when they cannot validate or convert data structures + due to invalid format, missing required fields, or type mismatches. + Contains the original validation error details for debugging. + + Usage: + Typically caught when loading configuration files or processing + external data that doesn't conform to UTCP specifications. + """ diff --git a/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py b/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py index 10d9573..3e31104 100644 --- a/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py +++ b/core/src/utcp/implementations/post_processors/filter_dict_post_processor.py @@ -10,6 +10,22 @@ from utcp.utcp_client import UtcpClient class FilterDictPostProcessor(ToolPostProcessor): + """REQUIRED + Post-processor that filters dictionary keys from tool results. + + Provides flexible filtering capabilities to include or exclude specific keys + from dictionary results, with support for nested dictionaries and lists. + Can be configured to apply filtering only to specific tools or manuals. + + Attributes: + tool_post_processor_type: Always "filter_dict" for this processor. + exclude_keys: List of keys to remove from dictionary results. + only_include_keys: List of keys to keep in dictionary results (all others removed). + exclude_tools: List of tool names to skip processing for. + only_include_tools: List of tool names to process (all others skipped). + exclude_manuals: List of manual names to skip processing for. + only_include_manuals: List of manual names to process (all others skipped). + """ tool_post_processor_type: Literal["filter_dict"] = "filter_dict" exclude_keys: Optional[List[str]] = None only_include_keys: Optional[List[str]] = None @@ -89,6 +105,8 @@ def _filter_dict_only_include_keys(self, result: Any) -> Any: return result class FilterDictPostProcessorConfigSerializer(Serializer[FilterDictPostProcessor]): + """REQUIRED + Serializer for FilterDictPostProcessor configuration.""" def to_dict(self, obj: FilterDictPostProcessor) -> dict: return obj.model_dump() diff --git a/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py b/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py index c0a19a1..da28eb8 100644 --- a/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py +++ b/core/src/utcp/implementations/post_processors/limit_strings_post_processor.py @@ -10,6 +10,22 @@ from utcp.utcp_client import UtcpClient class LimitStringsPostProcessor(ToolPostProcessor): + """REQUIRED + Post-processor that limits the length of string values in tool results. + + Truncates string values to a specified maximum length to prevent + excessively large responses. Processes nested dictionaries and lists + recursively. Can be configured to apply limiting only to specific + tools or manuals. + + Attributes: + tool_post_processor_type: Always "limit_strings" for this processor. + limit: Maximum length for string values (default: 10000 characters). + exclude_tools: List of tool names to skip processing for. + only_include_tools: List of tool names to process (all others skipped). + exclude_manuals: List of manual names to skip processing for. + only_include_manuals: List of manual names to process (all others skipped). + """ tool_post_processor_type: Literal["limit_strings"] = "limit_strings" limit: int = 10000 exclude_tools: Optional[List[str]] = None @@ -39,6 +55,8 @@ def _process_object(self, obj: Any) -> Any: return obj class LimitStringsPostProcessorConfigSerializer(Serializer[LimitStringsPostProcessor]): + """REQUIRED + Serializer for LimitStringsPostProcessor configuration.""" def to_dict(self, obj: LimitStringsPostProcessor) -> dict: return obj.model_dump() diff --git a/core/src/utcp/implementations/tag_search.py b/core/src/utcp/implementations/tag_search.py index d258c63..f11bf40 100644 --- a/core/src/utcp/implementations/tag_search.py +++ b/core/src/utcp/implementations/tag_search.py @@ -9,7 +9,24 @@ class TagAndDescriptionWordMatchStrategy(ToolSearchStrategy): """REQUIRED Tag and description word match strategy. - This strategy matches tools based on the presence of tags and words in the description. + Implements a weighted scoring system that matches tools based on: + 1. Tag matches (higher weight) + 2. Description word matches (lower weight) + + The strategy normalizes queries to lowercase, extracts words using regex, + and calculates relevance scores for each tool. Results are sorted by + score in descending order. + + Attributes: + tool_search_strategy_type: Always "tag_and_description_word_match". + description_weight: Weight multiplier for description word matches (default: 1.0). + tag_weight: Weight multiplier for tag matches (default: 3.0). + + Scoring Algorithm: + - Each matching tag contributes tag_weight points + - Each matching description word contributes description_weight points + - Tools with higher scores are ranked first + - Tools with zero score are included in results (ranked last) """ tool_search_strategy_type: Literal["tag_and_description_word_match"] = "tag_and_description_word_match" description_weight: float = 1 diff --git a/core/src/utcp/plugins/plugin_loader.py b/core/src/utcp/plugins/plugin_loader.py index 18b6b0b..4666f1f 100644 --- a/core/src/utcp/plugins/plugin_loader.py +++ b/core/src/utcp/plugins/plugin_loader.py @@ -1,6 +1,16 @@ import importlib.metadata def _load_plugins(): + """REQUIRED + Load and register all built-in and external UTCP plugins. + + Registers core serializers for authentication, variable loading, tool repositories, + search strategies, and post-processors. Also discovers and loads external plugins + through the 'utcp.plugins' entry point group. + + This function is called automatically by ensure_plugins_initialized() and should + not be called directly. + """ from utcp.plugins.discovery import register_auth, register_variable_loader, register_tool_repository, register_tool_search_strategy, register_tool_post_processor from utcp.interfaces.concurrent_tool_repository import ConcurrentToolRepositoryConfigSerializer from utcp.interfaces.tool_search_strategy import ToolSearchStrategyConfigSerializer diff --git a/plugins/communication_protocols/cli/README.md b/plugins/communication_protocols/cli/README.md index 8febb5a..246493c 100644 --- a/plugins/communication_protocols/cli/README.md +++ b/plugins/communication_protocols/cli/README.md @@ -1 +1,162 @@ -Find the UTCP readme at https://github.com/universal-tool-calling-protocol/python-utcp. \ No newline at end of file +# UTCP CLI Plugin + +[![PyPI Downloads](https://static.pepy.tech/badge/utcp-cli)](https://pepy.tech/projects/utcp-cli) + +Command-line interface plugin for UTCP, enabling integration with command-line tools and processes. + +## Features + +- **Command Execution**: Run any command-line tool as a UTCP tool +- **Environment Variables**: Secure credential and configuration passing +- **Working Directory Control**: Execute commands in specific directories +- **Input/Output Handling**: Support for stdin, stdout, stderr processing +- **Cross-Platform**: Works on Windows, macOS, and Linux +- **Timeout Management**: Configurable execution timeouts +- **Argument Validation**: Optional input sanitization + +## Installation + +```bash +pip install utcp-cli +``` + +## Quick Start + +```python +from utcp.utcp_client import UtcpClient + +# Basic CLI tool +client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "file_tools", + "call_template_type": "cli", + "command_name": "ls -la ${path}" + }] +}) + +result = await client.call_tool("file_tools.list", {"path": "/home"}) +``` + +## Configuration Examples + +### Basic Command +```json +{ + "name": "file_ops", + "call_template_type": "cli", + "command_name": "ls -la ${path}", + "working_dir": "/tmp" +} +``` + +### With Environment Variables +```json +{ + "name": "python_script", + "call_template_type": "cli", + "command_name": "python script.py ${input}", + "env_vars": { + "PYTHONPATH": "/custom/path", + "API_KEY": "${API_KEY}" + } +} +``` + +### Processing JSON with jq +```json +{ + "name": "json_processor", + "call_template_type": "cli", + "command_name": "jq '.data'", + "stdin": "${json_input}", + "timeout": 10 +} +``` + +### Git Operations +```json +{ + "name": "git_tools", + "call_template_type": "cli", + "command_name": "git ${operation} ${args}", + "working_dir": "${repo_path}", + "env_vars": { + "GIT_AUTHOR_NAME": "${author_name}", + "GIT_AUTHOR_EMAIL": "${author_email}" + } +} +``` + +## Security Considerations + +- Commands run in isolated subprocesses +- Environment variables provide secure credential passing +- Working directory restrictions limit file system access +- Input validation prevents command injection + +```json +{ + "name": "safe_grep", + "call_template_type": "cli", + "command_name": "grep ${pattern} ${file}", + "working_dir": "/safe/directory", + "allowed_args": { + "pattern": "^[a-zA-Z0-9_-]+$", + "file": "^[a-zA-Z0-9_./-]+\\.txt$" + } +} +``` + +## Error Handling + +```python +from utcp.exceptions import ToolCallError +import subprocess + +try: + result = await client.call_tool("cli_tool.command", {"arg": "value"}) +except ToolCallError as e: + if isinstance(e.__cause__, subprocess.CalledProcessError): + print(f"Command failed with exit code {e.__cause__.returncode}") + print(f"stderr: {e.__cause__.stderr}") +``` + +## Common Use Cases + +- **File Operations**: ls, find, grep, awk, sed +- **Data Processing**: jq, sort, uniq, cut +- **System Monitoring**: ps, top, df, netstat +- **Development Tools**: git, npm, pip, docker +- **Custom Scripts**: Python, bash, PowerShell scripts + +## Testing CLI Tools + +```python +import pytest +from utcp.utcp_client import UtcpClient + +@pytest.mark.asyncio +async def test_cli_tool(): + client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "test_cli", + "call_template_type": "cli", + "command_name": "echo ${message}" + }] + }) + + result = await client.call_tool("test_cli.echo", {"message": "hello"}) + assert "hello" in result["stdout"] +``` + +## Related Documentation + +- [Main UTCP Documentation](../../../README.md) +- [Core Package Documentation](../../../core/README.md) +- [HTTP Plugin](../http/README.md) +- [MCP Plugin](../mcp/README.md) +- [Text Plugin](../text/README.md) + +## Examples + +For complete examples, see the [UTCP examples repository](https://github.com/universal-tool-calling-protocol/utcp-examples). diff --git a/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py b/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py index 3d83508..fb4badf 100644 --- a/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py +++ b/plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py @@ -8,17 +8,54 @@ class CliCallTemplate(CallTemplate): """REQUIRED - Call template configuration for Command Line Interface tools. + Call template configuration for Command Line Interface (CLI) tools. - Enables execution of command-line tools and programs as UTCP providers. - Supports environment variable injection and custom working directories. + This class defines the configuration for executing command-line tools and + programs as UTCP tool providers. It supports environment variable injection, + custom working directories, and defines the command to be executed. Attributes: - call_template_type: Always "cli" for CLI providers. - command_name: The name or path of the command to execute. - env_vars: Optional environment variables to set during command execution. - working_dir: Optional custom working directory for command execution. - auth: Always None - CLI providers don't support authentication. + call_template_type: The type of the call template. Must be "cli". + command_name: The command or path of the program to execute. It can + contain placeholders for arguments that will be substituted at + runtime (e.g., `${arg_name}`). + env_vars: A dictionary of environment variables to set for the command's + execution context. Values can be static strings or placeholders for + variables from the UTCP client's variable substitutor. + working_dir: The working directory from which to run the command. If not + provided, it defaults to the current process's working directory. + auth: Authentication details. Not applicable to the CLI protocol, so it + is always None. + + Examples: + Basic CLI command: + ```json + { + "name": "list_files_tool", + "call_template_type": "cli", + "command_name": "ls -la", + "working_dir": "/tmp" + } + ``` + + Command with environment variables and argument placeholders: + ```json + { + "name": "python_script_tool", + "call_template_type": "cli", + "command_name": "python script.py --input ${input_file}", + "env_vars": { + "PYTHONPATH": "/custom/path", + "API_KEY": "${API_KEY_VAR}" + } + } + ``` + + Security Considerations: + - Commands are executed in a subprocess. Ensure that the commands + specified are from a trusted source. + - Avoid passing unsanitized user input directly into the command string. + Use tool argument validation where possible. """ call_template_type: Literal["cli"] = "cli" @@ -34,16 +71,39 @@ class CliCallTemplate(CallTemplate): class CliCallTemplateSerializer(Serializer[CliCallTemplate]): """REQUIRED - Serializer for CliCallTemplate.""" + Serializer for converting between `CliCallTemplate` and dictionary representations. + + This class handles the serialization and deserialization of `CliCallTemplate` + objects, ensuring that they can be correctly represented as dictionaries and + reconstructed from them, with validation. + """ def to_dict(self, obj: CliCallTemplate) -> dict: """REQUIRED - Converts a CliCallTemplate to a dictionary.""" + Converts a `CliCallTemplate` instance to its dictionary representation. + + Args: + obj: The `CliCallTemplate` instance to serialize. + + Returns: + A dictionary representing the `CliCallTemplate`. + """ return obj.model_dump() def validate_dict(self, obj: dict) -> CliCallTemplate: """REQUIRED - Validates a dictionary and returns a CliCallTemplate.""" + Validates a dictionary and constructs a `CliCallTemplate` instance. + + Args: + obj: The dictionary to validate and deserialize. + + Returns: + A `CliCallTemplate` instance. + + Raises: + UtcpSerializerValidationError: If the dictionary is not a valid + representation of a `CliCallTemplate`. + """ try: return CliCallTemplate.model_validate(obj) except Exception as e: diff --git a/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py b/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py index 148e261..0eecc8d 100644 --- a/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py +++ b/plugins/communication_protocols/cli/src/utcp_cli/cli_communication_protocol.py @@ -1,23 +1,21 @@ -"""Command Line Interface (CLI) transport for UTCP client. +"""Command Line Interface (CLI) communication protocol for the UTCP client. -This module provides the CLI transport implementation that enables UTCP clients -to interact with command-line tools and processes. It handles tool discovery -through startup commands, tool execution with proper argument formatting, -and output processing with JSON parsing capabilities. +This module provides an implementation of the `CommunicationProtocol` interface +that enables the UTCP client to interact with command-line tools. It supports +discovering tools by executing a command and parsing its output for a UTCP +manual, as well as calling those tools with arguments. Key Features: - - Asynchronous command execution with timeout handling - - Tool discovery via startup commands that output UTCP manuals - - Flexible argument formatting for command-line flags - - Environment variable support for authentication and configuration - - JSON output parsing with fallback to raw text - - Cross-platform command parsing (Windows/Unix) - - Working directory control for command execution - -Security: - - Command execution is isolated through subprocess - - Environment variables can be controlled per provider - - Working directory can be restricted + - Asynchronous execution of shell commands. + - Tool discovery by running a command that outputs a UTCP manual. + - Flexible argument formatting for different CLI conventions. + - Support for environment variables and custom working directories. + - Automatic parsing of JSON output with a fallback to raw text. + - Cross-platform command parsing for Windows and Unix-like systems. + +Security Considerations: + Executing arbitrary command-line tools can be dangerous. This protocol + should only be used with trusted tools. """ import asyncio import json @@ -38,33 +36,17 @@ class CliCommunicationProtocol(CommunicationProtocol): """REQUIRED - Transport implementation for CLI-based tool providers. - - Handles communication with command-line tools by executing processes - and managing their input/output. Supports both tool discovery and - execution phases with comprehensive error handling and timeout management. - - Features: - - Asynchronous subprocess execution with proper cleanup - - Tool discovery through startup commands returning UTCP manuals - - Flexible argument formatting for various CLI conventions - - Environment variable injection for authentication - - JSON output parsing with graceful fallback to text - - Cross-platform command parsing and execution - - Configurable working directories and timeouts - - Process lifecycle management with proper termination - - Architecture: - CLI tools are discovered by executing the provider's command_name - and parsing the output for UTCP manual JSON. Tool calls execute - the same command with formatted arguments and return processed output. - - Attributes: - _log: Logger function for debugging and error reporting. + Communication protocol for interacting with CLI-based tool providers. + + This class implements the `CommunicationProtocol` interface to handle + communication with command-line tools. It discovers tools by executing a + command specified in a `CliCallTemplate` and parsing the output for a UTCP + manual. It also executes tool calls by running the corresponding command + with the provided arguments. """ def __init__(self): - """Initialize the CLI transport.""" + """Initializes the `CliCommunicationProtocol`.""" def _log_info(self, message: str): """Log informational messages.""" @@ -156,9 +138,24 @@ async def _execute_command( async def register_manual(self, caller, manual_call_template: CallTemplate) -> RegisterManualResult: """REQUIRED - Register a CLI manual and discover its tools. - - Executes the call template's command_name and looks for a UTCP manual JSON in the output. + Registers a CLI-based manual and discovers its tools. + + This method executes the command specified in the `CliCallTemplate`'s + `command_name` field. It then attempts to parse the command's output + (stdout) as a UTCP manual in JSON format. + + Args: + caller: The UTCP client instance that is calling this method. + manual_call_template: The `CliCallTemplate` containing the details for + tool discovery, such as the command to run. + + Returns: + A `RegisterManualResult` object indicating whether the registration + was successful and containing the discovered tools. + + Raises: + ValueError: If the `manual_call_template` is not an instance of + `CliCallTemplate` or if `command_name` is not set. """ if not isinstance(manual_call_template, CliCallTemplate): raise ValueError("CliCommunicationProtocol can only be used with CliCallTemplate") @@ -240,7 +237,15 @@ async def register_manual(self, caller, manual_call_template: CallTemplate) -> R async def deregister_manual(self, caller, manual_call_template: CallTemplate) -> None: """REQUIRED - Deregister a CLI manual (no-op).""" + Deregisters a CLI manual. + + For the CLI protocol, this is a no-op as there are no persistent + connections to terminate. + + Args: + caller: The UTCP client instance that is calling this method. + manual_call_template: The call template of the manual to deregister. + """ if isinstance(manual_call_template, CliCallTemplate): self._log_info( f"Deregistering CLI manual '{manual_call_template.name}' (no-op)" @@ -403,23 +408,27 @@ def _parse_tool_data(self, data: Any, provider_name: str) -> List[Tool]: async def call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> Any: """REQUIRED - Call a CLI tool. - - Executes the command specified by provider.command_name with the provided arguments. - + Calls a CLI tool by executing its command. + + This method constructs and executes the command specified in the + `CliCallTemplate`. It formats the provided `tool_args` as command-line + arguments and runs the command in a subprocess. + Args: - caller: The UTCP client that is calling this method. - tool_name: Name of the tool to call - tool_args: Arguments for the tool call - tool_call_template: The CliCallTemplate for the tool - + caller: The UTCP client instance that is calling this method. + tool_name: The name of the tool to call. + tool_args: A dictionary of arguments for the tool call. + tool_call_template: The `CliCallTemplate` for the tool. + Returns: - The output from the command execution based on exit code: - - If exit code is 0: stdout (parsed as JSON if possible, otherwise raw string) - - If exit code is not 0: stderr - + The result of the command execution. If the command exits with a code + of 0, it returns the content of stdout. If the exit code is non-zero, + it returns the content of stderr. The output is parsed as JSON if + possible; otherwise, it is returned as a raw string. + Raises: - ValueError: If provider is not a CliProvider or command_name is not set + ValueError: If `tool_call_template` is not an instance of + `CliCallTemplate` or if `command_name` is not set. """ if not isinstance(tool_call_template, CliCallTemplate): raise ValueError("CliCommunicationProtocol can only be used with CliCallTemplate") @@ -476,13 +485,9 @@ async def call_tool(self, caller, tool_name: str, tool_args: Dict[str, Any], too async def call_tool_streaming(self, caller, tool_name: str, tool_args: Dict[str, Any], tool_call_template: CallTemplate) -> AsyncGenerator[Any, None]: """REQUIRED - Streaming calls are not supported for CLI protocol.""" - raise NotImplementedError("Streaming is not supported by the CLI communication protocol.") - - async def close(self) -> None: - """ - Close the transport. - - This is a no-op for CLI transports since they don't maintain connections. + Streaming calls are not supported for the CLI protocol. + + Raises: + NotImplementedError: Always, as this functionality is not supported. """ - self._log_info("Closing CLI transport (no-op)") + raise NotImplementedError("Streaming is not supported by the CLI communication protocol.") diff --git a/plugins/communication_protocols/http/README.md b/plugins/communication_protocols/http/README.md index 8febb5a..5f66bb2 100644 --- a/plugins/communication_protocols/http/README.md +++ b/plugins/communication_protocols/http/README.md @@ -1 +1,144 @@ -Find the UTCP readme at https://github.com/universal-tool-calling-protocol/python-utcp. \ No newline at end of file +# UTCP HTTP Plugin + +[![PyPI Downloads](https://static.pepy.tech/badge/utcp-http)](https://pepy.tech/projects/utcp-http) + +HTTP communication protocol plugin for UTCP, supporting REST APIs, Server-Sent Events (SSE), and streaming HTTP. + +## Features + +- **HTTP/REST APIs**: Full support for GET, POST, PUT, DELETE, PATCH methods +- **Authentication**: API key, Basic Auth, OAuth2 support +- **Server-Sent Events (SSE)**: Real-time event streaming +- **Streaming HTTP**: Large response handling with chunked transfer +- **OpenAPI Integration**: Automatic tool generation from OpenAPI specs +- **Path Parameters**: URL templating with `{parameter}` syntax +- **Custom Headers**: Static and dynamic header support + +## Installation + +```bash +pip install utcp-http +``` + +## Quick Start + +```python +from utcp.utcp_client import UtcpClient + +# Basic HTTP API +client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "api_service", + "call_template_type": "http", + "url": "https://api.example.com/users/{user_id}", + "http_method": "GET" + }] +}) + +result = await client.call_tool("api_service.get_user", {"user_id": "123"}) +``` + +## Configuration Examples + +### Basic HTTP Request +```json +{ + "name": "my_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "http_method": "GET" +} +``` + +### With API Key Authentication +```json +{ + "name": "secure_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "http_method": "POST", + "auth": { + "auth_type": "api_key", + "api_key": "${API_KEY}", + "var_name": "X-API-Key", + "location": "header" + } +} +``` + +### OAuth2 Authentication +```json +{ + "name": "oauth_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } +} +``` + +### Server-Sent Events (SSE) +```json +{ + "name": "event_stream", + "call_template_type": "sse", + "url": "https://api.example.com/events", + "event_type": "message", + "reconnect": true +} +``` + +### Streaming HTTP +```json +{ + "name": "large_data", + "call_template_type": "streamable_http", + "url": "https://api.example.com/download", + "chunk_size": 8192 +} +``` + +## OpenAPI Integration + +Automatically generate UTCP tools from OpenAPI specifications: + +```python +from utcp_http.openapi_converter import OpenApiConverter + +converter = OpenApiConverter() +manual = await converter.convert_openapi_to_manual( + "https://api.example.com/openapi.json" +) + +client = await UtcpClient.create() +await client.register_manual(manual) +``` + +## Error Handling + +```python +from utcp.exceptions import ToolCallError +import httpx + +try: + result = await client.call_tool("api.get_data", {"id": "123"}) +except ToolCallError as e: + if isinstance(e.__cause__, httpx.HTTPStatusError): + print(f"HTTP {e.__cause__.response.status_code}: {e.__cause__.response.text}") +``` + +## Related Documentation + +- [Main UTCP Documentation](../../../README.md) +- [Core Package Documentation](../../../core/README.md) +- [CLI Plugin](../cli/README.md) +- [MCP Plugin](../mcp/README.md) +- [Text Plugin](../text/README.md) + +## Examples + +For complete examples, see the [UTCP examples repository](https://github.com/universal-tool-calling-protocol/utcp-examples). diff --git a/plugins/communication_protocols/http/src/utcp_http/http_call_template.py b/plugins/communication_protocols/http/src/utcp_http/http_call_template.py index 73e4d64..b3a9e70 100644 --- a/plugins/communication_protocols/http/src/utcp_http/http_call_template.py +++ b/plugins/communication_protocols/http/src/utcp_http/http_call_template.py @@ -15,6 +15,70 @@ class HttpCallTemplate(CallTemplate): parameters using {parameter_name} syntax. All tool arguments not mapped to URL body, headers or query pattern parameters are passed as query parameters using '?arg_name={arg_value}'. + Configuration Examples: + Basic HTTP GET request: + ```json + { + "name": "my_rest_api", + "call_template_type": "http", + "url": "https://api.example.com/users/{user_id}", + "http_method": "GET" + } + ``` + + POST with authentication: + ```json + { + "name": "secure_api", + "call_template_type": "http", + "url": "https://api.example.com/users", + "http_method": "POST", + "content_type": "application/json", + "auth": { + "auth_type": "api_key", + "api_key": "Bearer ${API_KEY}", + "var_name": "Authorization", + "location": "header" + }, + "headers": { + "X-Custom-Header": "value" + }, + "body_field": "body", + "header_fields": ["user_id"] + } + ``` + + OAuth2 authentication: + ```json + { + "name": "oauth_api", + "call_template_type": "http", + "url": "https://api.example.com/data", + "http_method": "GET", + "auth": { + "auth_type": "oauth2", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "token_url": "https://auth.example.com/token" + } + } + ``` + + Basic authentication: + ```json + { + "name": "basic_auth_api", + "call_template_type": "http", + "url": "https://api.example.com/secure", + "http_method": "GET", + "auth": { + "auth_type": "basic", + "username": "${USERNAME}", + "password": "${PASSWORD}" + } + } + ``` + Attributes: call_template_type: Always "http" for HTTP providers. http_method: The HTTP method to use for requests. diff --git a/plugins/communication_protocols/http/src/utcp_http/openapi_converter.py b/plugins/communication_protocols/http/src/utcp_http/openapi_converter.py index 1a6b679..be20fe6 100644 --- a/plugins/communication_protocols/http/src/utcp_http/openapi_converter.py +++ b/plugins/communication_protocols/http/src/utcp_http/openapi_converter.py @@ -5,13 +5,13 @@ mapping, and proper tool creation from REST API specifications. Key Features: - - OpenAPI 2.0 and 3.0 specification support - - Automatic JSON reference ($ref) resolution - - Authentication scheme mapping (API key, Basic, OAuth2) - - Input/output schema extraction from OpenAPI schemas - - URL path parameter handling - - Request body and header field mapping - - Provider name generation from specification metadata + - OpenAPI 2.0 and 3.0 specification support. + - Automatic JSON reference ($ref) resolution. + - Authentication scheme mapping (API key, Basic, OAuth2). + - Input/output schema extraction from OpenAPI schemas. + - URL path parameter handling. + - Request body and header field mapping. + - Call template name generation from specification metadata. The converter creates UTCP tools that can be used to interact with REST APIs defined by OpenAPI specifications, providing a bridge between OpenAPI and UTCP. @@ -38,14 +38,42 @@ class OpenApiConverter: a UTCP tool with appropriate input/output schemas. Features: - - Complete OpenAPI specification parsing - - Recursive JSON reference ($ref) resolution - - Authentication scheme conversion (API key, Basic, OAuth2) - - Input parameter and request body handling - - Response schema extraction - - URL template and path parameter support - - Provider name normalization - - Placeholder variable generation for configuration + - Complete OpenAPI specification parsing. + - Recursive JSON reference ($ref) resolution. + - Authentication scheme conversion (API key, Basic, OAuth2). + - Input parameter and request body handling. + - Response schema extraction. + - URL template and path parameter support. + - Call template name normalization. + - Placeholder variable generation for configuration. + + Usage Examples: + Basic OpenAPI conversion: + ```python + from utcp_http.openapi_converter import OpenApiConverter + + # Assuming you have a method to fetch and parse the spec + openapi_spec = fetch_and_parse_spec("https://api.example.com/openapi.json") + + converter = OpenApiConverter(openapi_spec) + manual = converter.convert() + + # Use the generated manual with a UTCP client + # client = await UtcpClient.create() + # await client.register_manual(manual) + ``` + + Converting local OpenAPI file: + ```python + import yaml + + converter = OpenApiConverter() + with open("api_spec.yaml", "r") as f: + spec_content = yaml.safe_load(f) + + converter = OpenApiConverter(spec_content) + manual = converter.convert() + ``` Architecture: The converter works by iterating through all paths and operations @@ -60,14 +88,14 @@ class OpenApiConverter: """ def __init__(self, openapi_spec: Dict[str, Any], spec_url: Optional[str] = None, call_template_name: Optional[str] = None): - """Initialize the OpenAPI converter. + """Initializes the OpenAPI converter. Args: openapi_spec: Parsed OpenAPI specification as a dictionary. spec_url: Optional URL where the specification was retrieved from. Used for base URL determination if servers are not specified. - call_template_name: Optional custom name for the call_template if - the specification title is not provided. + call_template_name: Optional custom name for the call_template if + the specification title is not provided. """ self.spec = openapi_spec self.spec_url = spec_url @@ -99,7 +127,15 @@ def _get_placeholder(self, placeholder_name: str) -> str: def convert(self) -> UtcpManual: """REQUIRED - Parses the OpenAPI specification and returns a UtcpManual.""" + Converts the loaded OpenAPI specification into a UtcpManual. + + This is the main entry point for the conversion process. It iterates through + the paths and operations in the specification, creating a UTCP tool for each + one. + + Returns: + A UtcpManual object containing all the tools generated from the spec. + """ self.placeholder_counter = 0 tools = [] servers = self.spec.get("servers") diff --git a/plugins/communication_protocols/mcp/README.md b/plugins/communication_protocols/mcp/README.md index 8febb5a..0aa06f4 100644 --- a/plugins/communication_protocols/mcp/README.md +++ b/plugins/communication_protocols/mcp/README.md @@ -1 +1,253 @@ -Find the UTCP readme at https://github.com/universal-tool-calling-protocol/python-utcp. \ No newline at end of file +# UTCP MCP Plugin + +[![PyPI Downloads](https://static.pepy.tech/badge/utcp-mcp)](https://pepy.tech/projects/utcp-mcp) + +Model Context Protocol (MCP) interoperability plugin for UTCP, enabling seamless integration with existing MCP servers. + +## Features + +- **MCP Server Integration**: Connect to existing MCP servers +- **Stdio Transport**: Local process-based MCP servers +- **HTTP Transport**: Remote MCP server connections +- **OAuth2 Authentication**: Secure authentication for HTTP servers +- **Migration Support**: Gradual migration from MCP to UTCP +- **Tool Discovery**: Automatic tool enumeration from MCP servers +- **Session Management**: Efficient connection handling + +## Installation + +```bash +pip install utcp-mcp +``` + +## Quick Start + +```python +from utcp.utcp_client import UtcpClient + +# Connect to MCP server +client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "mcp_server", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["mcp-server.js"] + } + } + } + }] +}) + +# Call MCP tool through UTCP +result = await client.call_tool("mcp_server.filesystem.read_file", { + "path": "/data/file.txt" +}) +``` + +## Configuration Examples + +### Stdio Transport (Local Process) +```json +{ + "name": "local_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "python", + "args": ["-m", "mcp_filesystem_server"], + "env": {"LOG_LEVEL": "INFO"} + } + } + } +} +``` + +### HTTP Transport (Remote Server) +```json +{ + "name": "remote_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "api_server": { + "transport": "http", + "url": "https://mcp.example.com" + } + } + } +} +``` + +### With OAuth2 Authentication +```json +{ + "name": "secure_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "secure_server": { + "transport": "http", + "url": "https://mcp.example.com" + } + } + }, + "auth": { + "auth_type": "oauth2", + "token_url": "https://auth.example.com/token", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "scope": "read:tools" + } +} +``` + +### Multiple MCP Servers +```json +{ + "name": "multi_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "python", + "args": ["-m", "mcp_filesystem"] + }, + "database": { + "command": "node", + "args": ["mcp-db-server.js"], + "cwd": "/app/mcp-servers" + } + } + } +} +``` + +## Migration Scenarios + +### Gradual Migration from MCP to UTCP + +**Phase 1: MCP Integration** +```python +# Use existing MCP servers through UTCP +client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "legacy_mcp", + "call_template_type": "mcp", + "config": {"mcpServers": {"server": {...}}} + }] +}) +``` + +**Phase 2: Mixed Environment** +```python +# Mix MCP and native UTCP tools +client = await UtcpClient.create(config={ + "manual_call_templates": [ + { + "name": "legacy_mcp", + "call_template_type": "mcp", + "config": {"mcpServers": {"old_server": {...}}} + }, + { + "name": "new_api", + "call_template_type": "http", + "url": "https://api.example.com/utcp" + } + ] +}) +``` + +**Phase 3: Full UTCP** +```python +# Pure UTCP implementation +client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "native_utcp", + "call_template_type": "http", + "url": "https://api.example.com/utcp" + }] +}) +``` + +## Debugging and Troubleshooting + +### Enable Debug Logging +```python +import logging +logging.getLogger('utcp.mcp').setLevel(logging.DEBUG) + +try: + client = await UtcpClient.create(config=mcp_config) + tools = await client.list_tools() +except TimeoutError: + print("MCP server connection timed out") +``` + +### List Available Tools +```python +# Discover tools from MCP server +tools = await client.list_tools() +print(f"Available tools: {[tool.name for tool in tools]}") +``` + +### Connection Testing +```python +@pytest.mark.asyncio +async def test_mcp_integration(): + client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "test_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "test": { + "command": "python", + "args": ["-m", "test_mcp_server"] + } + } + } + }] + }) + + tools = await client.list_tools() + assert len(tools) > 0 + + result = await client.call_tool("test_mcp.echo", {"message": "test"}) + assert result["message"] == "test" +``` + +## Error Handling + +```python +from utcp.exceptions import ToolCallError + +try: + result = await client.call_tool("mcp_server.tool", {"arg": "value"}) +except ToolCallError as e: + print(f"MCP tool call failed: {e}") + # Check if it's a connection issue, authentication error, etc. +``` + +## Performance Considerations + +- **Session Reuse**: MCP plugin reuses connections when possible +- **Timeout Configuration**: Set appropriate timeouts for MCP operations +- **Resource Cleanup**: Sessions are automatically cleaned up +- **Concurrent Calls**: Multiple tools can be called concurrently + +## Related Documentation + +- [Main UTCP Documentation](../../../README.md) +- [Core Package Documentation](../../../core/README.md) +- [HTTP Plugin](../http/README.md) +- [CLI Plugin](../cli/README.md) +- [Text Plugin](../text/README.md) +- [MCP Specification](https://modelcontextprotocol.io/) + +## Examples + +For complete examples, see the [UTCP examples repository](https://github.com/universal-tool-calling-protocol/utcp-examples). diff --git a/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py b/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py index 5755804..90b3329 100644 --- a/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py +++ b/plugins/communication_protocols/mcp/src/utcp_mcp/mcp_call_template.py @@ -37,6 +37,87 @@ class McpCallTemplate(CallTemplate): interfaces. Supports both stdio (local process) and HTTP (remote) transport methods. + Configuration Examples: + Basic MCP server with stdio transport: + ```json + { + "name": "mcp_server", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "filesystem": { + "command": "node", + "args": ["mcp-server.js"], + "env": {"NODE_ENV": "production"} + } + } + } + } + ``` + + MCP server with working directory: + ```json + { + "name": "mcp_tools", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "tools": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/app/mcp", + "env": { + "PYTHONPATH": "/app", + "LOG_LEVEL": "INFO" + } + } + } + } + } + ``` + + MCP server with OAuth2 authentication: + ```json + { + "name": "secure_mcp", + "call_template_type": "mcp", + "config": { + "mcpServers": { + "secure_server": { + "transport": "http", + "url": "https://mcp.example.com" + } + } + }, + "auth": { + "auth_type": "oauth2", + "token_url": "https://auth.example.com/token", + "client_id": "${CLIENT_ID}", + "client_secret": "${CLIENT_SECRET}", + "scope": "read:tools" + } + } + ``` + + Migration Examples: + During migration (UTCP with MCP): + ```python + # UTCP Client with MCP plugin + client = await UtcpClient.create() + result = await client.call_tool("filesystem.read_file", { + "path": "/data/file.txt" + }) + ``` + + After migration (Pure UTCP): + ```python + # UTCP Client with native protocol + client = await UtcpClient.create() + result = await client.call_tool("filesystem.read_file", { + "path": "/data/file.txt" + }) + ``` + Attributes: call_template_type: Always "mcp" for MCP providers. config: Configuration object containing MCP server definitions. diff --git a/plugins/communication_protocols/text/README.md b/plugins/communication_protocols/text/README.md index 8febb5a..53a7fd3 100644 --- a/plugins/communication_protocols/text/README.md +++ b/plugins/communication_protocols/text/README.md @@ -1 +1,126 @@ -Find the UTCP readme at https://github.com/universal-tool-calling-protocol/python-utcp. \ No newline at end of file +# UTCP Text Plugin + +[![PyPI Downloads](https://static.pepy.tech/badge/utcp-text)](https://pepy.tech/projects/utcp-text) + +A simple, file-based resource plugin for UTCP. This plugin allows you to define tools that return the content of a specified local file. + +## Features + +- **Local File Content**: Define tools that read and return the content of local files. +- **UTCP Manual Discovery**: Load tool definitions from local UTCP manual files in JSON or YAML format. +- **Static & Simple**: Ideal for returning mock data, configuration, or any static text content from a file. +- **Version Control**: Tool definitions and their corresponding content files can be versioned with your code. +- **No Authentication**: Designed for simple, local file access without authentication. + +## Installation + +```bash +pip install utcp-text +``` + +## How It Works + +The Text plugin operates in two main ways: + +1. **Tool Discovery (`register_manual`)**: It can read a standard UTCP manual file (e.g., `my-tools.json`) to learn about available tools. This is how the `UtcpClient` discovers what tools can be called. +2. **Tool Execution (`call_tool`)**: When you call a tool, the plugin looks at the `tool_call_template` associated with that tool. It expects a `text` template, and it will read and return the entire content of the `file_path` specified in that template. + +**Important**: The `call_tool` function **does not** use the arguments you pass to it. It simply returns the full content of the file defined in the tool's template. + +## Quick Start + +Here is a complete example demonstrating how to define and use a tool that returns the content of a file. + +### 1. Create a Content File + +First, create a file with some content that you want your tool to return. + +`./mock_data/user.json`: +```json +{ + "id": 123, + "name": "John Doe", + "email": "john.doe@example.com" +} +``` + +### 2. Create a UTCP Manual + +Next, define a UTCP manual that describes your tool. The `tool_call_template` must be of type `text` and point to the content file you just created. + +`./manuals/local_tools.json`: +```json +{ + "manual_version": "1.0.0", + "utcp_version": "1.0.1", + "tools": [ + { + "name": "get_mock_user", + "description": "Returns a mock user profile from a local file.", + "tool_call_template": { + "call_template_type": "text", + "file_path": "./mock_data/user.json" + } + } + ] +} +``` + +### 3. Use the Tool in Python + +Finally, use the `UtcpClient` to load the manual and call the tool. + +```python +import asyncio +from utcp.utcp_client import UtcpClient + +async def main(): + # Create a client, providing the path to the manual. + # The text plugin is used automatically for the "text" call_template_type. + client = await UtcpClient.create(config={ + "manual_call_templates": [{ + "name": "local_file_tools", + "call_template_type": "text", + "file_path": "./manuals/local_tools.json" + }] + }) + + # List the tools to confirm it was loaded + tools = await client.list_tools() + print("Available tools:", [tool.name for tool in tools]) + + # Call the tool. The result will be the content of './mock_data/user.json' + result = await client.call_tool("local_file_tools.get_mock_user", {}) + + print("\nTool Result:") + print(result) + +if __name__ == "__main__": + asyncio.run(main()) +``` + +### Expected Output: + +``` +Available tools: ['local_file_tools.get_mock_user'] + +Tool Result: +{ + "id": 123, + "name": "John Doe", + "email": "john.doe@example.com" +} +``` + +## Use Cases + +- **Mocking**: Return mock data for tests or local development without needing a live server. +- **Configuration**: Load static configuration files as tool outputs. +- **Templates**: Retrieve text templates (e.g., for emails or reports). + +## Related Documentation + +- [Main UTCP Documentation](../../../README.md) +- [Core Package Documentation](../../../core/README.md) +- [HTTP Plugin](../http/README.md) - For calling real web APIs. +- [CLI Plugin](../cli/README.md) - For executing command-line tools.