Skip to content

Commit ff219bf

Browse files
improve docs and section formatting
1 parent 0b0cfc0 commit ff219bf

File tree

4 files changed

+168
-110
lines changed

4 files changed

+168
-110
lines changed

en/developer-docs/docs/develop-components/sharing-and-reusing/create-a-connection-to-a-genai-service.md

Lines changed: 37 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,29 +2,49 @@
22

33
## Overview
44

5-
Choreo is a powerful platform that enables developers to create, deploy, and consume services efficiently. The Choreo Internal Marketplace simplifies API discovery and usage, allowing developers to integrate APIs seamlessly into their applications.
5+
Choreo is a cloud-native platform that enables developers to create, deploy, and consume services efficiently. The **Choreo Internal Marketplace** streamlines API discovery and integration, making it easier to incorporate services into your applications.
66

7-
This guide is intended for application developers who wish to consume Gen AI Services published in the Internal Marketplace to build their applications.
7+
This guide is for application developers who want to consume GenAI Services published in the Internal Marketplace.
88

99
## Prerequisites
1010

11-
Before proceeding, ensure you have access to a published Gen AI Service to consume. If you do not have one, please follow the PE documentation to create and deploy a sample Gen AI Service.
11+
Before you begin, ensure that:
1212

13-
## Discover Gen AI Services
13+
- You have access to a published GenAI Service in the Internal Marketplace.
1414

15-
1. Navigate to the [Choreo Internal Marketplace](https://console.choreo.dev) and sign in.
16-
2. Select the desired Gen AI Service to view its details.
15+
- If you don’t yet have one, refer to the [PE documentation](https://docs.choreo.dev/en/pe-docs/docs/db-and-services/ai-gateway/integrate-and-manage-gen-ai-services) to create and deploy a sample GenAI Service.
1716

18-
## Create a connection to a Gen AI Service
17+
## Discover GenAI Services
1918

20-
Follow these steps to create a connection to a Gen AI Service from your choreo component:
19+
1. Navigate to the [Choreo Internal Marketplace](https://console.choreo.dev).
2120

22-
1. In the left navigation menu, click **Connections**. This page lists all the existing connections.
23-
2. Click **+Create**. This opens the Marketplace view where you can browse and search for services.
24-
3. Click the **Services** tab. You can search and apply filters to efficiently find a service.
25-
4. Click on the Gen AI service you want to connect to.
26-
5. Enter a name and a description for the connection.
27-
6. Select an **Access Mode** and **Authentication Scheme** for the connection.
28-
7. Click **Create**.
29-
30-
This creates the connection and displays its details for each environment, along with an inline guide on how to use the connection in your component.
21+
2. Sign in with your credentials.
22+
3. Browse or search for the desired GenAI Service.
23+
24+
4. Select the service to view its details.
25+
26+
## Create a Connection to a GenAI Service
27+
28+
To use a GenAI Service in your Choreo component, you first need to create a connection.
29+
30+
1. In the left navigation menu, click Connections.
31+
32+
2. This page lists all existing connections.
33+
34+
3. Click **+Create**.
35+
36+
4. The Marketplace view opens, where you can browse and search for services.
37+
38+
4. Click the **Services** tab.
39+
40+
5. Use search or filters to find the required service efficiently.
41+
42+
6. Select the GenAI Service you want to connect to.
43+
44+
7. Provide a name and description for the connection.
45+
46+
8. Choose an Access Mode and Authentication Scheme.
47+
48+
9. Click **Create**.
49+
50+
Once created, the connection details are displayed for each environment. An inline usage guide is also provided, showing how to use the connection in your component.
281 KB
Loading
Lines changed: 38 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,45 @@
11
# Choreo AI Gateway
22

3-
As AI adoption accelerates, managing AI APIs effectively has become essential for organizations integrating AI into their applications. Choreo's AI Gateway simplifies this process, providing a seamless way to create, manage, and expose AI APIs with robust security and scalability.
3+
As organizations accelerate AI adoption, managing AI APIs effectively has become essential. The Choreo AI Gateway simplifies this process by providing a unified way to create, manage, and expose AI APIs with enterprise-grade security, observability, and scalability.
44

5-
![AI Gateway](../../assets/img/ai-gateway/ai-gateway.png)
5+
![Choreo AI Gateway](../../assets/img/ai-gateway/ai-gateway.png)
66

7-
Choreo offers built-in support for OpenAI, Azure OpenAI, and Mistral and AWS Bedrock.
7+
## Supported Providers
88

9-
With a comprehensive set of capabilities, the AI Gateway ensures secure and efficient AI integration. Key features include:
9+
The AI Gateway integrates seamlessly with leading AI platforms, including:
1010

11-
- AI Vendor Key Configuration: Securely authenticate AI APIs by configuring API keys obtained from the AI vendor.
12-
- Rate Limiting: Protect AI backends by enforcing token-based rate limits to manage resource consumption.
13-
- AI API Observability: Track AI API usage statistics using Analytics solutions. (coming soon)
11+
- OpenAI
1412

15-
By leveraging these capabilities, organizations can efficiently integrate, monitor, and scale AI APIs, unlocking the full potential of AI-driven applications.
13+
- Azure OpenAI
14+
15+
- Anthropic Claude
16+
17+
- Mistral
18+
19+
- AWS Bedrock
20+
21+
## Key Features
22+
23+
The AI Gateway offers a robust set of capabilities to ensure secure and efficient AI integration:
24+
25+
- AI Vendor Key Configuration:
26+
27+
Securely authenticate with AI providers by configuring API keys obtained from each vendor.
28+
29+
- Token-Based Rate Limiting:
30+
31+
Protect backend services and manage costs by applying rate limits based on token usage rather than just request counts.
32+
33+
- AI API Observability (Coming Soon):
34+
35+
Gain insights into AI API performance and usage with built-in analytics and monitoring.
36+
37+
With these features, organizations can integrate, monitor, and scale AI services efficiently, unlocking the full potential of AI-driven applications.
38+
39+
### Token-Based Rate Limiting
40+
41+
AI services typically charge based on tokens rather than requests, making cost control critical. Choreo AI Gateway enables token-based rate limiting at the API level to:
42+
43+
- Define quotas based on LLM token consumption.
44+
- Prevent unexpected costs by controlling overuse.
45+
- Enforce fair usage policies across AI-powered APIs.
Lines changed: 93 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -1,150 +1,158 @@
11
# Integrate and Manage Generative AI Services
22

3-
Generative AI (GenAI) services leverage advanced machine learning models to create original content such as text, images, music, or code, by identifying and learning patterns from existing data. Powered by deep neural networks and other machine learning models, these services can generate human-like outputs in various formats. This versatility makes GenAI ideal for tasks such as content creation, image generation, conversation automation, etc.
3+
## Overview
44

5-
Choreo provides native support to integrate GenAI services into applications. Developers can register GenAI services, manage access, and expose them securely through the [Choreo AI Gateway](https://github.com/wso2/choreo-ai-gateway), ensuring enterprise-grade governance and control.
5+
Generative AI (GenAI) services leverage advanced machine learning models to produce original content—such as text, images, music, or code—by learning patterns from existing data. Powered by deep neural networks, these services can generate human-like outputs across multiple formats, making them well-suited for tasks like content creation, image generation, and conversational automation.
6+
7+
Choreo provides native support for integrating GenAI services into applications. Using the Choreo AI Gateway, Platform Engineers and Admins can:
8+
9+
- Register GenAI services in the Choreo AI Gateway.
10+
11+
- Manage access and authentication for GenAI services.
12+
13+
- Enforce governance, security, and cost controls for GenAI services at scale.
614

715
## Register a GenAI Service
816

9-
When you register a GenAI service in Choreo, all invocations are automatically routed through the Choreo AI Gateway. The gateway enforces:
10-
- Security – Authentication, authorization, and secure transport.
11-
- Rate limiting – Request- and token-based throttling.
12-
- Observability – Metrics, logging, and monitoring for all requests (coming soon)
17+
When you register a GenAI service in Choreo, all requests are routed through the AI Gateway, which enforces:
18+
19+
- Security – Authentication, authorization, and encrypted transport.
20+
21+
- Rate Limiting – Token-based throttling.
1322

14-
This ensures consistent, secure, and cost-controlled access to the underlying GenAI provider.
23+
- Observability (Coming Soon) – Metrics, logging, and monitoring.
1524

16-
### Scopes of Registration
17-
GenAI services can be registered at two scopes:
25+
This ensures consistent, secure, and cost-controlled access to the underlying provider.
1826

19-
- **Organization-level registration**: The service is available across all projects within the organization.
20-
- **Project-level registration**: The service is restricted to the specific project where it is registered.
27+
## Registration Scopes
28+
29+
You can register GenAI services at two levels:
30+
31+
- Organization-level registration – Service is available across all projects in the organization.
32+
33+
- Project-level registration – Service is restricted to the project where it is registered.
2134

2235
Once registered, the service is published to the Internal Marketplace, where developers can discover and consume it via Connections.
2336

24-
### Prerequisites
25-
Before registering a GenAI service, obtain the following details from the provider:
37+
## Prerequisites
38+
39+
Before registering a GenAI service, obtain the following from your provider:
2640

2741
- API key
42+
2843
- Service URL
44+
2945
- Any additional credentials (e.g., client secrets, subscription keys)
3046

3147
!!! note
32-
- Register services at the correct scope:
33-
- Use organization-level if the service will be shared across projects.
48+
49+
- Choose the correct registration scope:
50+
51+
- Use organization-level if the service will be shared across multiple projects.
3452
- Use project-level if usage is limited to a single project.
3553

54+
## Registration Steps
55+
3656
### Step 1: Select a Service Provider
3757

38-
1. Sign in to the [Choreo Console](https://console.choreo.dev).
39-
2. Select your Organization or Project depending on the required scope.
40-
3. In the left navigation menu, go to **DB & Services → GenAI Services**.
41-
4. Select a service provider:
42-
- **OpenAI**
43-
- **Azure OpenAI**
44-
- **Anthropic Claude**
45-
- **Mistral**
46-
- **AWS Bedrock**
58+
1. Sign in to the Choreo Console.
59+
60+
2. Select your Organization or Project, depending on the required scope.
61+
62+
3. In the left navigation menu, go to DB & Services → GenAI Services.
63+
64+
4. Choose a provider from the list:
65+
66+
- OpenAI
67+
68+
- Azure OpenAI
4769

48-
5. Click **Next**.
70+
- Anthropic Claude
71+
72+
- Mistral
73+
74+
- AWS Bedrock
75+
76+
5. Click Next.
4977

5078
### Step 2: Provide Service Details
5179

52-
1. Under **Register a GenAI Service**, enter the following details:
80+
Under **Register a GenAI Service**, enter:
81+
82+
- Name – The service name.
83+
84+
- Context Path – A unique identifier for the API within the AI Gateway.
85+
86+
- Service URL – The provider’s endpoint.
5387

54-
- **Name**: The name of the service.
55-
- **Context path**: A unique identifier for the service API within the Choreo AI Gateway.
56-
- **Service URL**: The provider’s endpoint.
57-
- **Summary**: A short description of the service.
88+
- Summary – A short description of the service.
5889

59-
2. Click **Next**.
90+
Click **Next**.
6091

6192
### Step 3: Add Environment Configurations
6293

63-
Under **Add Environment Configurations**, configure backend security and token-based rate limiting for your AI API.
94+
![Add Environment Configurations](../../assets/img/ai-gateway/add-configs.png)
6495

65-
!!! note
66-
- Required configurations vary by provider.
67-
- You must provide valid configurations for all environments to complete registration.
96+
### Configure Backend Security
6897

69-
#### Configure Backend Security
98+
In **Backend Security**, set the Authorization header.
7099

71-
1. In the AI API Security section:
100+
- Provide the API key or credentials (e.g., Authorization: Bearer <token>).
72101

73-
- Set the Authorization header.
74-
- Provide the API key or credentials from your provider (e.g., `Authorization: Bearer <token>`).
75-
- Key formats may differ by provider.
102+
- Key formats may vary by provider.
76103

77-
![Add environment configurations](../../assets/img/ai-gateway/add-configs.png)
104+
### Configure Token-Based Rate Limiting
78105

79-
#### Configure Token-Based Rate Limiting
106+
LLMs and GenAI services are billed based on token usage, not just request count. Token-based rate limiting helps:
80107

81-
Unlike traditional APIs, LLMs and GenAI services are billed based on token consumption rather than just request count. Token-based rate limiting allows you to:
82108
- Control costs.
109+
83110
- Prevent excessive usage.
111+
84112
- Enforce fair quotas across consumers.
85113

86-
2. In the **Token-Based Rate Limiting** section, configure the following parameters:
87-
- **Time Unit**: Window for quota enforcement (per minute, per hour, per day).
88-
- **Max Prompt Token Count**: Limit for input tokens per request.
89-
- **Max Completion Token Count**: Limit for output tokens per request.
90-
- **Max Total Token Count**: Combined input + output tokens per request.
114+
Set the following parameters:
91115

92-
3. Click **Register**.
116+
- Time Unit – Enforcement window (per minute, hour, or day).
93117

94-
The service is now available in the Internal Marketplace.
118+
- Max Prompt Token Count – Maximum input tokens per request.
95119

96-
### View or Update Service Details
120+
- Max Completion Token Count – Maximum output tokens per request.
97121

98-
1. Sign in to the [Choreo Console](https://console.choreo.dev).
99-
2. Navigate to **DB & Services → GenAI Services**.
100-
3. Select a service to view or edit its details:
122+
- Max Total Token Count – Combined input + output tokens per request.
101123

102-
- **General Details**: Displays metadata such as name, description, and labels..
103-
- **Service Definition**: Displays the service definition.
104-
- **Environment Configurations**: Displays the security and rate-limiting settings for each environment, along with deployment status.
124+
Click **Register**.
105125

106-
![View or update service details](../../assets/img/ai-gateway/view-configs.png)
126+
The service is now available in the Internal Marketplace, where developers can discover and connect to it.
107127

108-
### Add a GenAI Service to the Internal Marketplace
128+
See the [Developer Docs](https://wso2.com/choreo/docs/develop-components/sharing-and-reusing/create-a-connection-to-a-genai-service/) for instructions on how to consume a GenAI service published in the Internal Marketplace.
109129

110-
1. Sign in to the [Choreo Console](https://console.choreo.dev).
111-
2. Navigate to **DB & Services → GenAI Services**.
112-
3. Select the service.
113-
4. Click **Add to Marketplace**.
130+
## Manage Registered Services
114131

115-
### Remove a GenAI Service from the Internal Marketplace
132+
### View or Update Service Details
116133

117134
1. Sign in to the [Choreo Console](https://console.choreo.dev).
118-
2. Navigate to **DB & Services → GenAI Services**.
119-
3. Select the service.
120-
4. Click **Remove from Marketplace**.
121135

122-
The service is removed from the Internal Marketplace, making it unavailable for new Connections. However, existing Connections continue to function until manually removed.
136+
2. Navigate to DB & Services → GenAI Services.
123137

124-
## Manage GenAI Services
138+
3. Select a service to view or edit details:
125139

126-
### View or Update Service Details
140+
- General Details – Name, description, and labels.
127141

128-
1. Sign in to the [Choreo Console](https://console.choreo.dev).
129-
2. Navigate to **DB & Services → GenAI Services**.
130-
3. Select a service to view or edit its details:
142+
- Environment Configurations – Backend security and rate limiting.
143+
144+
- Deployment Status – View deployment state for each environment.
131145

132-
- **General Details**: Displays service metadata. For example, the service name, overview, labels, etc.
133-
- **Service Definition**: Displays the service definition. To update the definition, click **Upload** and select the new definition file.
134-
- **Environment Configurations**: Displays the environment configurations for the service. To update the configurations, expand the environment and update the configurations. You can also view the deployment status of the AI API in the environment.
146+
![Deployment Status](../../assets/img/ai-gateway/view-configs.png)
135147

136-
### Add a GenAI Service to the Internal Marketplace
148+
### Remove a GenAI Service
137149

138150
1. Sign in to the [Choreo Console](https://console.choreo.dev).
139-
2. Navigate to **DB & Services → GenAI Services**.
140-
3. Select the service.
141-
4. Click **Add to Marketplace**.
142151

143-
### Remove a GenAI Service from the Internal Marketplace
152+
2. Navigate to DB & Services → GenAI Services.
144153

145-
1. Sign in to the [Choreo Console](https://console.choreo.dev).
146-
2. Navigate to **DB & Services → GenAI Services**.
147154
3. Select the service.
148-
4. Click **Remove from Marketplace**.
149155

150-
The service is removed from the Internal Marketplace, making it unavailable for new Connections. However, existing Connections continue to function until manually removed.
156+
4. Click Remove from Marketplace.
157+
158+
The service is removed from the Internal Marketplace and can’t be used for new Connections. Existing Connections remain functional until manually removed.

0 commit comments

Comments
 (0)