|
1 | 1 | # Integrate and Manage Generative AI Services |
2 | 2 |
|
3 | | -Generative AI (GenAI) services leverage advanced machine learning models to create original content such as text, images, music, or code, by identifying and learning patterns from existing data. Powered by deep neural networks and other machine learning models, these services can generate human-like outputs in various formats. This versatility makes GenAI ideal for tasks such as content creation, image generation, conversation automation, etc. |
| 3 | +## Overview |
4 | 4 |
|
5 | | -Choreo provides native support to integrate GenAI services into applications. Developers can register GenAI services, manage access, and expose them securely through the [Choreo AI Gateway](https://github.com/wso2/choreo-ai-gateway), ensuring enterprise-grade governance and control. |
| 5 | +Generative AI (GenAI) services leverage advanced machine learning models to produce original content—such as text, images, music, or code—by learning patterns from existing data. Powered by deep neural networks, these services can generate human-like outputs across multiple formats, making them well-suited for tasks like content creation, image generation, and conversational automation. |
| 6 | + |
| 7 | +Choreo provides native support for integrating GenAI services into applications. Using the Choreo AI Gateway, Platform Engineers and Admins can: |
| 8 | + |
| 9 | +- Register GenAI services in the Choreo AI Gateway. |
| 10 | + |
| 11 | +- Manage access and authentication for GenAI services. |
| 12 | + |
| 13 | +- Enforce governance, security, and cost controls for GenAI services at scale. |
6 | 14 |
|
7 | 15 | ## Register a GenAI Service |
8 | 16 |
|
9 | | -When you register a GenAI service in Choreo, all invocations are automatically routed through the Choreo AI Gateway. The gateway enforces: |
10 | | -- Security – Authentication, authorization, and secure transport. |
11 | | -- Rate limiting – Request- and token-based throttling. |
12 | | -- Observability – Metrics, logging, and monitoring for all requests (coming soon) |
| 17 | +When you register a GenAI service in Choreo, all requests are routed through the AI Gateway, which enforces: |
| 18 | + |
| 19 | +- Security – Authentication, authorization, and encrypted transport. |
| 20 | + |
| 21 | +- Rate Limiting – Token-based throttling. |
13 | 22 |
|
14 | | -This ensures consistent, secure, and cost-controlled access to the underlying GenAI provider. |
| 23 | +- Observability (Coming Soon) – Metrics, logging, and monitoring. |
15 | 24 |
|
16 | | -### Scopes of Registration |
17 | | -GenAI services can be registered at two scopes: |
| 25 | +This ensures consistent, secure, and cost-controlled access to the underlying provider. |
18 | 26 |
|
19 | | -- **Organization-level registration**: The service is available across all projects within the organization. |
20 | | -- **Project-level registration**: The service is restricted to the specific project where it is registered. |
| 27 | +## Registration Scopes |
| 28 | + |
| 29 | +You can register GenAI services at two levels: |
| 30 | + |
| 31 | +- Organization-level registration – Service is available across all projects in the organization. |
| 32 | + |
| 33 | +- Project-level registration – Service is restricted to the project where it is registered. |
21 | 34 |
|
22 | 35 | Once registered, the service is published to the Internal Marketplace, where developers can discover and consume it via Connections. |
23 | 36 |
|
24 | | -### Prerequisites |
25 | | -Before registering a GenAI service, obtain the following details from the provider: |
| 37 | +## Prerequisites |
| 38 | + |
| 39 | +Before registering a GenAI service, obtain the following from your provider: |
26 | 40 |
|
27 | 41 | - API key |
| 42 | + |
28 | 43 | - Service URL |
| 44 | + |
29 | 45 | - Any additional credentials (e.g., client secrets, subscription keys) |
30 | 46 |
|
31 | 47 | !!! note |
32 | | - - Register services at the correct scope: |
33 | | - - Use organization-level if the service will be shared across projects. |
| 48 | + |
| 49 | + - Choose the correct registration scope: |
| 50 | + |
| 51 | + - Use organization-level if the service will be shared across multiple projects. |
34 | 52 | - Use project-level if usage is limited to a single project. |
35 | 53 |
|
| 54 | +## Registration Steps |
| 55 | + |
36 | 56 | ### Step 1: Select a Service Provider |
37 | 57 |
|
38 | | -1. Sign in to the [Choreo Console](https://console.choreo.dev). |
39 | | -2. Select your Organization or Project depending on the required scope. |
40 | | -3. In the left navigation menu, go to **DB & Services → GenAI Services**. |
41 | | -4. Select a service provider: |
42 | | - - **OpenAI** |
43 | | - - **Azure OpenAI** |
44 | | - - **Anthropic Claude** |
45 | | - - **Mistral** |
46 | | - - **AWS Bedrock** |
| 58 | +1. Sign in to the Choreo Console. |
| 59 | + |
| 60 | +2. Select your Organization or Project, depending on the required scope. |
| 61 | + |
| 62 | +3. In the left navigation menu, go to DB & Services → GenAI Services. |
| 63 | + |
| 64 | +4. Choose a provider from the list: |
| 65 | + |
| 66 | +- OpenAI |
| 67 | + |
| 68 | +- Azure OpenAI |
47 | 69 |
|
48 | | -5. Click **Next**. |
| 70 | +- Anthropic Claude |
| 71 | + |
| 72 | +- Mistral |
| 73 | + |
| 74 | +- AWS Bedrock |
| 75 | + |
| 76 | +5. Click Next. |
49 | 77 |
|
50 | 78 | ### Step 2: Provide Service Details |
51 | 79 |
|
52 | | -1. Under **Register a GenAI Service**, enter the following details: |
| 80 | +Under **Register a GenAI Service**, enter: |
| 81 | + |
| 82 | +- Name – The service name. |
| 83 | + |
| 84 | +- Context Path – A unique identifier for the API within the AI Gateway. |
| 85 | + |
| 86 | +- Service URL – The provider’s endpoint. |
53 | 87 |
|
54 | | - - **Name**: The name of the service. |
55 | | - - **Context path**: A unique identifier for the service API within the Choreo AI Gateway. |
56 | | - - **Service URL**: The provider’s endpoint. |
57 | | - - **Summary**: A short description of the service. |
| 88 | +- Summary – A short description of the service. |
58 | 89 |
|
59 | | -2. Click **Next**. |
| 90 | +Click **Next**. |
60 | 91 |
|
61 | 92 | ### Step 3: Add Environment Configurations |
62 | 93 |
|
63 | | -Under **Add Environment Configurations**, configure backend security and token-based rate limiting for your AI API. |
| 94 | + |
64 | 95 |
|
65 | | -!!! note |
66 | | -- Required configurations vary by provider. |
67 | | -- You must provide valid configurations for all environments to complete registration. |
| 96 | +### Configure Backend Security |
68 | 97 |
|
69 | | -#### Configure Backend Security |
| 98 | +In **Backend Security**, set the Authorization header. |
70 | 99 |
|
71 | | -1. In the AI API Security section: |
| 100 | +- Provide the API key or credentials (e.g., Authorization: Bearer <token>). |
72 | 101 |
|
73 | | - - Set the Authorization header. |
74 | | - - Provide the API key or credentials from your provider (e.g., `Authorization: Bearer <token>`). |
75 | | - - Key formats may differ by provider. |
| 102 | +- Key formats may vary by provider. |
76 | 103 |
|
77 | | -  |
| 104 | +### Configure Token-Based Rate Limiting |
78 | 105 |
|
79 | | -#### Configure Token-Based Rate Limiting |
| 106 | +LLMs and GenAI services are billed based on token usage, not just request count. Token-based rate limiting helps: |
80 | 107 |
|
81 | | -Unlike traditional APIs, LLMs and GenAI services are billed based on token consumption rather than just request count. Token-based rate limiting allows you to: |
82 | 108 | - Control costs. |
| 109 | + |
83 | 110 | - Prevent excessive usage. |
| 111 | + |
84 | 112 | - Enforce fair quotas across consumers. |
85 | 113 |
|
86 | | -2. In the **Token-Based Rate Limiting** section, configure the following parameters: |
87 | | - - **Time Unit**: Window for quota enforcement (per minute, per hour, per day). |
88 | | - - **Max Prompt Token Count**: Limit for input tokens per request. |
89 | | - - **Max Completion Token Count**: Limit for output tokens per request. |
90 | | - - **Max Total Token Count**: Combined input + output tokens per request. |
| 114 | +Set the following parameters: |
91 | 115 |
|
92 | | -3. Click **Register**. |
| 116 | +- Time Unit – Enforcement window (per minute, hour, or day). |
93 | 117 |
|
94 | | -The service is now available in the Internal Marketplace. |
| 118 | +- Max Prompt Token Count – Maximum input tokens per request. |
95 | 119 |
|
96 | | -### View or Update Service Details |
| 120 | +- Max Completion Token Count – Maximum output tokens per request. |
97 | 121 |
|
98 | | -1. Sign in to the [Choreo Console](https://console.choreo.dev). |
99 | | -2. Navigate to **DB & Services → GenAI Services**. |
100 | | -3. Select a service to view or edit its details: |
| 122 | +- Max Total Token Count – Combined input + output tokens per request. |
101 | 123 |
|
102 | | - - **General Details**: Displays metadata such as name, description, and labels.. |
103 | | - - **Service Definition**: Displays the service definition. |
104 | | - - **Environment Configurations**: Displays the security and rate-limiting settings for each environment, along with deployment status. |
| 124 | +Click **Register**. |
105 | 125 |
|
106 | | -  |
| 126 | +The service is now available in the Internal Marketplace, where developers can discover and connect to it. |
107 | 127 |
|
108 | | -### Add a GenAI Service to the Internal Marketplace |
| 128 | +See the [Developer Docs](https://wso2.com/choreo/docs/develop-components/sharing-and-reusing/create-a-connection-to-a-genai-service/) for instructions on how to consume a GenAI service published in the Internal Marketplace. |
109 | 129 |
|
110 | | -1. Sign in to the [Choreo Console](https://console.choreo.dev). |
111 | | -2. Navigate to **DB & Services → GenAI Services**. |
112 | | -3. Select the service. |
113 | | -4. Click **Add to Marketplace**. |
| 130 | +## Manage Registered Services |
114 | 131 |
|
115 | | -### Remove a GenAI Service from the Internal Marketplace |
| 132 | +### View or Update Service Details |
116 | 133 |
|
117 | 134 | 1. Sign in to the [Choreo Console](https://console.choreo.dev). |
118 | | -2. Navigate to **DB & Services → GenAI Services**. |
119 | | -3. Select the service. |
120 | | -4. Click **Remove from Marketplace**. |
121 | 135 |
|
122 | | -The service is removed from the Internal Marketplace, making it unavailable for new Connections. However, existing Connections continue to function until manually removed. |
| 136 | +2. Navigate to DB & Services → GenAI Services. |
123 | 137 |
|
124 | | -## Manage GenAI Services |
| 138 | +3. Select a service to view or edit details: |
125 | 139 |
|
126 | | -### View or Update Service Details |
| 140 | +- General Details – Name, description, and labels. |
127 | 141 |
|
128 | | -1. Sign in to the [Choreo Console](https://console.choreo.dev). |
129 | | -2. Navigate to **DB & Services → GenAI Services**. |
130 | | -3. Select a service to view or edit its details: |
| 142 | +- Environment Configurations – Backend security and rate limiting. |
| 143 | + |
| 144 | +- Deployment Status – View deployment state for each environment. |
131 | 145 |
|
132 | | - - **General Details**: Displays service metadata. For example, the service name, overview, labels, etc. |
133 | | - - **Service Definition**: Displays the service definition. To update the definition, click **Upload** and select the new definition file. |
134 | | - - **Environment Configurations**: Displays the environment configurations for the service. To update the configurations, expand the environment and update the configurations. You can also view the deployment status of the AI API in the environment. |
| 146 | + |
135 | 147 |
|
136 | | -### Add a GenAI Service to the Internal Marketplace |
| 148 | +### Remove a GenAI Service |
137 | 149 |
|
138 | 150 | 1. Sign in to the [Choreo Console](https://console.choreo.dev). |
139 | | -2. Navigate to **DB & Services → GenAI Services**. |
140 | | -3. Select the service. |
141 | | -4. Click **Add to Marketplace**. |
142 | 151 |
|
143 | | -### Remove a GenAI Service from the Internal Marketplace |
| 152 | +2. Navigate to DB & Services → GenAI Services. |
144 | 153 |
|
145 | | -1. Sign in to the [Choreo Console](https://console.choreo.dev). |
146 | | -2. Navigate to **DB & Services → GenAI Services**. |
147 | 154 | 3. Select the service. |
148 | | -4. Click **Remove from Marketplace**. |
149 | 155 |
|
150 | | -The service is removed from the Internal Marketplace, making it unavailable for new Connections. However, existing Connections continue to function until manually removed. |
| 156 | +4. Click Remove from Marketplace. |
| 157 | + |
| 158 | +The service is removed from the Internal Marketplace and can’t be used for new Connections. Existing Connections remain functional until manually removed. |
0 commit comments